What Crowdsourcing Can Offer to Cross-Cultural Psychological Science

Abstract

Although the benefits of crowdsourcing research models have been outlined elsewhere, very little attention has been paid to the application of these models to cross-cultural behavioral research. In this manuscript, we delineate two types of crowdsourcing initiatives—researcher crowdsourced and participant crowdsourced. Researcher crowdsourced refers to initiatives where researchers are gathered to work toward a shared goal. Participant crowdsourced refers to those which allow a researcher to gather a large number of participants within a short time frame. We explore the utility of each type of initiative while providing readers with a framework that can be used when deciding whether researcher or participant crowdsourcing initiatives would be most fruitful for their work. Perceived strengths of a researcher crowdsourced initiative with a cross-cultural focus is based on contributor data from Psi Chi’s Network for International Collaborative Exchange (NICE) and is integrated into this framework. Claims are made for the utility of both researcher and participant crowdsourcing as a way to increase generalizability and reliability, decrease time burdens, democratize research, educate individuals on open science, and provide mentorship. These claims are supported with data from NICE contributors.

Keywords

crowdsourcing cross-cultural metascience methodology psychology

Traditional research within psychological science used to be conducted primarily by a single researcher, or a small group of researchers, who answer a research question using a convenience sample (Merton, 1968). Although this traditional research approach offers several advantages (e.g., more personal, first-authored publications, a quicker research process, convenience, etc.) there are several significant drawbacks. Notably—traditional approaches to research are particularly detrimental to cross-cultural researchers who seek to understand the influence of cultural variations on human behavior. For example, one barrier to cross-cultural work that may be exacerbated when working alone or with a small group of researchers is the difficulty of obtaining samples from various cultural groups (Jameson, 1994). In addition, researchers might need physical proximity to perform certain experiments, or linguistic expertise to converse with people living outside the researcher’s local context. Furthermore, checking one’s assumptions and biases about the population one is studying becomes increasingly difficult without working as part of a large multi-cultural team. Collaborative projects, specifically researcher and participant crowdsourced projects, can provide an avenue for circumventing many of these barriers to conducting cross-cultural research.

Several manuscripts have focused on the benefits that various forms of crowdsourcing can offer to the validity of psychological science (Uhlmann et al., 2019), and to the researchers who use these various crowdsourcing initiatives (Miller et al., 2017). Despite the increasing popularity of researcher and participant crowdsourcing, very little work has focused on the utility of such methods for conducting cross-cultural behavioral research (Gelfand et al., 2011 for exceptions). Researchers can better utilize resources to fit their needs if they understand the utility and benefits of both researcher and participant crowdsourcing models in relation to cross-cultural research. Furthermore, while there is a plethora of published literature detailing perceptions of, and best practices for using participant crowdsourced initiatives such as Amazon Mechanical Turk (mTurk; Hauser et al., 2018) the field is lacking an understanding of collaborators’ attitudes toward participating in researcher crowdsourced data collection. Understanding how research collaborators perceive their participation in researcher crowdsourced data collection can inform the development of future projects as well as their implementation (such as highlighting benefits when advertising opportunities to participate in conducting a study).

This manuscript seeks to achieve the following aims: (1) highlight the utility of researcher and participant crowdsourced initiatives for cross-cultural research, (2) provide a framework that cross-cultural behavioral researchers can use when selecting a researcher or participant crowdsourcing model, (3) detail perceived strengths of using researcher and participant crowdsourced approaches in cross-cultural behavioral research, and (4) describe perceptions of researcher crowdsourcing from collaborators who participated in a specific cross-cultural psychology research project.

Aim 1: Crowdsourcing Initiatives

Overview of Crowdsourcing

The term crowdsourcing has two parts, crowd, which indicates a large number of people, and sourcing, a verb meaning “to obtain.” In the context of social science research, crowdsourcing can typically be conceptualized as the development of a network of individuals working toward a common goal which is organized by central key personnel, or an organizing body. To delineate, an individual or organizing body may crowdsource researchers and/or participants to achieve specific goals regarding concerns over scientific quality and feasibility. For instance, an example of a crowdsourcing model where researchers are crowdsourced is the Many Labs series (Ebersole et al., 2016; Klein et al., 2014, 2018). Here, multiple researchers are crowdsourced as collaborators to evaluate the replicability of psychological findings across different labs. This method mobilizes a variety of researchers, using their diverse resources and specific areas of expertise, to accomplish a shared goal (Handler & Conill, 2016; Uhlmann et al., 2019). In such models, responsibility and ownership of the research is diffused across a larger number of individuals; researchers making up the “crowd” usually have specific tasks customized to their interests and abilities. This method helps to ensure that individuals’ strengths and weaknesses are appropriately balanced. As such, we define researcher crowdsourced initiatives as those in which a number of researchers, typically of diverse geographical location (Moshontz et al., 2018 for an example) and skill sets, are brought together in order to achieve a research goal that could not be achieved alone.

An example of a crowdsourcing model where participants are crowdsourced is Amazon Mechanical Turk (mTurk; Harms & DeSimone, 2015). Here, a single researcher or lab can recruit participants for their research projects. This platform, and similar ones, afford researchers some control to increase the demographic diversity of their sample and their statistical power. In exchange for their participation researchers must compensate the participants; this amount is typically based on how long the survey took to complete. Additionally, researchers must also pay mTurk a fee based on how much the participants are being paid and how many are being recruited (at the time of writing, the typical fee paid to Amazon is 40% of what is being paid to the participant). Thus, participant crowdsourced initiatives are conceptualized here as those that allow researchers to obtain a large sample of participants with varying demographic characteristics within a relatively short time-frame.

While not every initiative has the same explicit goals, crowdsourcing, broadly speaking, has been seen positively in the social sciences due to favorable outcomes such as: demographic diversity of researchers and participants, high statistical power, increased generalizability of information, and the promotion of open science practices. Many crowdsourcing initiatives that aimed to source both researchers and participants were posed as beneficial for the social sciences because of their ability to address qualms regarding statistical inferences, generalizability, and replicability.

These efforts resemble citizen science, the engagement of the general public in research projects that often address salient issues in the community, large scale societal issues (e.g., Alzheimer’s) or require coordinated efforts (BrightFocus Foundation, 2018; Del Savio et al., 2016; Wiggins & Crowston, 2011). Citizen science represents a form of crowdsourcing that engages the general public and varies along multiple dimensions including scale, emphasis on civic education, level of citizen engagement, and project goals (Eitzel et al., 2017). As such, while citizen science represents an important form of crowdsourcing, reviewing the ability of citizen science to aid cross-cultural research is outside the scope of this manuscript. For a review of citizen science typologies, as well as strengths and critiques of these approaches readers are urged to see Dickinson et al., (2010), Eitzel et al. (2017) and Wiggins and Crowston (2011). Broadly speaking, both researcher and participant crowdsourced initiatives are collaborative efforts by which individuals work toward a common goal; collaborators may work at different levels or on specific tasks, but all collaborators are necessary to make the project successful. There are two distinct types of crowdsourced initiatives that will be reviewed here: researcher crowdsourced and participant crowdsourced. In the succeeding section, we will highlight these projects’ utility for cross-cultural research.

Researcher Crowdsourced

There are a few notable researcher crowdsourced initiatives, such as the Collaborative Replications and Education Project (CREP; Grahe et al., 2014; Wagge et al., 2018), the Pipeline Projects (Schweinsberg et al., 2016), the Psychological Science Accelerator (PSA; Moshontz et al., 2018) and Psi Chi’s Network for International Collaborative Exchange: Crowd component (NICE: Crowd) which source researchers to replicate previously-published work or yet-to-be published work through organization from a central body/researcher. These projects are important to note because they are easily accessible and afford methodological benefits for cross-cultural researchers. Such benefits include: a prespecified study design, a team of researchers that you can draw upon for specific advice and questions, and a study that is less likely to engage in questionable research practices than when researchers are working alone (John et al., 2012; Uhlmann et al., 2019). Each projects’ methodology, goals, and utility to cross-cultural researchers are described in detail in Supplementary Table 1.

Participant Crowdsourced

Another option for crowdsourcing is to crowdsource participants, rather than researchers or labs. Typically, the crowdsourcing of participants is achieved by automating the recruitment process via electronic methods and platforms (Palmer & Strickland, 2016). In these particular instances, internet services are used to invite a diverse population of participants to participate in research opportunities. Some examples of such methods include: Prolific, Qualtrics Panels, The StudyResponse Project, and Amazon Mechanical Turk. These initiatives allow researchers to tailor their participant recruitment based on demographic factors, and the use of remote recruitment (via the internet and third-party services) allow for increased sample size and diversity. Additionally, with these initiatives researchers have increased control over the research design and process, and subsequent timeline for research productivity. Each projects’ methodology, goals, and utility to cross-cultural researchers are described in detail in the second half of Supplementary Table 1.

Furthermore, participant crowdsourcing initiatives allow for cross-discipline collaborations in several fields. Shank (2016) notes the utility of mTurk for research in sociology. mTurk has also been noted as useful for marketing tasks and research, as well as biology research projects (Lee et al., 2017; Whitla, 2009). However, using electronic participant crowdsourcing platforms can be financially burdensome (Berinsky et al., 2012). As such, this financial restraint may impede researchers from getting the type of sample and/or the size of the sample that is required to answer their research question.

Summary

While each researcher crowdsourced initiative has its own specific goals, such initiatives afford collaborators with statistical power to conduct analyses, enhanced methodological rigor, scholarly activity, and resources for future collaborative projects. Host sites that facilitate participant crowdsourced initiatives (e.g., mTurk) often require payments to both participants and the host site which can make investigations costly. However, these initiatives broadly afford researchers flexibility in regards to both team size (researchers can collaborate, or work alone), and timeline (some researchers asking certain research questions can realistically complete a study on mTurk in less than a week and quickly post a follow up). Researchers can increase participants’ demographic diversity through use of additional services offered on sites that facilitate participant crowdsourcing (e.g., mTurk; Qualtrics). These services are aimed at increasing participant diversity or recruiting participants’ with specific characteristics. This is especially notable given that the use of convenience samples (typically undergraduate students) is a concern in many fields because it limits generalizability (Kam et al., 2007).

To review, participant crowdsourcing initiatives are beneficial for researchers who wish to retain some autonomy in the research process as participants are the ones being “sourced” to complete the task (i.e., the research). Further, these platforms often have a plethora of users (i.e., participants) available to complete the task quickly which makes web-based platforms convenient and time efficient. Researchers can also investigate a wide variety of research questions and utilize a wide variety of protocols via web-based platforms which allow for flexibility in research design and topics that can be investigated. The authors would also like to acknowledge that the initiatives reviewed here is not an exhaustive list, as many other initiatives exist, and many more are likely being developed. The reviewed initiatives are based on our familiarity with the projects, as well as reviews of the literature. Initiatives were also included only if they were still actively running projects so that readers could join such collaborative efforts.

Aim 2: Choosing a Crowdsourced Initiative

As highlighted above, different researcher and participant crowdsourcing initiatives have different benefits to offer to researchers. When considering which crowdsourcing initiative to choose, researchers should take these benefits into account while also considering other factors such as platforms for data collection, resources, collaborator engagement, sample sizes/power needed, and eligibility to co-author a manuscript.

Resources

As noted above, there are two types of crowdsourcing initiatives in which researchers can engage in: researcher crowdsourced or participant crowdsourced. All researcher crowdsourced initiatives reviewed above have no monetary fee associated with joining as a collaborator. Additionally, the majority of these projects adopt open science principles and seek to democratize the research process such that researchers are not inhibited by institutional status. Practically, this results in the sharing of protocols, materials, and data. While these projects typically have low financial cost, they often require a significant time investment (e.g., one academic year) as projects may have to go through proposal and revision stages (as is the case for the NICE: Crowd and PSA), and data must be collected across a significant number of sites. For researchers who have a participant pool available to them (e.g., undergraduate students) and a flexible timeline for research productivity, initiatives such as CREP, NICE: Crowd, Pipeline Projects, and PSA might be most appropriate. While these projects have a longer timeline (i.e., data collection period, data analysis period, manuscript writing period), participation is free, and researchers only need to contribute a specified number of cases for analysis.

Participant crowdsourced initiatives usually have a monetary fee associated with using their services as a researcher (e.g., mTurk; Qualtrics Panel). These initiatives, while time efficient, may not be monetarily sustainable for researchers unless they have access to funding or grant money. As such, researchers may need to reflect on the resources they have available, such as money, time, and their own participant pool.

If researchers have limited time, and restricted access to a participant pool but have the luxury of monetary funding, then participant crowdsourced initiatives may be more appropriate given the quick turnaround time for recruitment and data analysis. If all of a researcher’s resources are limited, it would be advisable for researchers to take into consideration other elements of both researcher and participant crowdsourcing initiatives such as the goal of the initiative, collaborator engagement, sample sizes/power, and eligibility for manuscript authorship. In addition, as a researcher, collecting data from a certain group of people may be another important consideration when selecting a platform to use for recruitment. If the researcher is interested in sourcing diverse groups of persons, generally, both participant crowdsourced platforms and researcher crowdsourced initiatives are equally plausible options. If a researcher is looking to target persons with very specific characteristics, participant crowdsourced platforms might be more applicable.

Engagement

Of the plethora of researcher crowdsourced projects available, many vary in the frequency and duration with which collaborators interact and communicate with each other, that is the extent to which collaborators engage. Depending on the researcher crowdsourced project’s structure, contributors may be in frequent communication with the project’s coordinator/inventor but have little to no contact with one another (e.g., Many Labs). In other researcher crowdsourcing initiatives (e.g., Pipeline Projects) team members are in frequent contact with each other in order to ensure fidelity to the research protocol and communicate findings. Researchers should reflect on what best suits their current needs, skill level, and collaboration style. For instance, more experienced researchers may feel comfortable working relatively independently while undergraduate and graduate students may want more support through the research process. Indeed, specific researcher crowdsourcing initiatives draw on the unharnessed power (and number) of undergraduate students (e.g., CREP). These initiatives, broadly speaking, help ensure that adequate quantities of data are collected by involving undergraduate students in the process. These projects also place undergraduate students in an active role within the research process, something that is critical to their academic development. The literature suggests that undergraduate research experience results in student gains such as increased ability to work independently, communication skills, critical thinking, and deepened knowledge of the field (Laursen et al., 2010).

Researchers may also approach researcher crowdsourced initiatives with a specific objective/need. For instance, if researchers are looking to expand their network of collaborators, a project in which collaborators are in frequent contact might better facilitate the development of interpersonal relationships. If researchers have the aim of learning more about researcher crowdsourcing, then a project that has a “contact person” or provides detailed instructions to collaborators might best suit this need. Additionally, if researchers are looking for feedback on a project or are seeking replications, then a project in which colleagues can critique and assess the study design would be ideal.

Open Science Elements

Notably, some researcher crowdsourced initiatives (e.g., NICE: Crowd, PSA) heavily utilize open science initiatives. Open science initiatives are gaining recognition, and many journals are adopting open science policies as part of the publication process (https://osf.io/4znzp/wiki/OSF%20for%20Journals/; McKiernan et al., 2016). The literature also supports open science initiatives as an avenue for improving the scientific quality of research (Asendorpf et al., 2013; Kidwell et al., 2016; Nosek et al., 2018), scientific efficiency, and democratizing knowledge (Arza & Fressoli, 2017). Given growing trends in the use of open science initiatives researchers interested in learning about these specific tools may benefit from having an “expert” to communicate with.

Sample Size and Power

One blanket benefit of both researcher and participant crowdsourcing initiatives is their ability to increase sample size and power. Because multiple persons are combining their participant pools and resources, or in the case of web-based participant crowdsourcing platforms on which a larger participant pool is available, individuals are able to achieve a considerable sample size for a particular study. Participant and researcher crowdsourcing initiatives may be particularly beneficial for researchers who have limited participant pools. Participant crowdsourced initiatives allows researchers to recruit hundreds of participants for a monetary cost, whereas researcher crowdsourced initiatives such as NICE: Crowd, CREP, or Many Labs only require that participants recruit a specified number of people to participate in the project. This reduces the strain of data collection for many researchers. NICE: Crowd in particular is beneficial for researchers with limited participant pools because (a) collaborators are only asked to contribute a specified number of cases, and (b) collaborators have access to the collaborative dataset which they can use to answer other previously-unaddressed research questions with adequate power. Adequate power is critical for reducing the risk of both Type I and Type II errors (Schweinsberg et al., 2016).

Manuscript Authorship

Different initiatives vary in how manuscript writing and authorship are approached. In participant crowdsourced initiatives, where a primary researcher is sourcing efforts from participants, or working with a small team, the manuscript is the sole responsibility of the selected individual(s) within the small team. For small teams working on these initiatives, authorship order is determined on a basis agreed upon by the authors. As the list of contributors grows, particularly within researcher crowdsourced initiatives, determining authorship order has the potential to become increasingly complicated. While determining authorship order and inclusion may seem like a challenge, collaborators can refer to the CRediT taxonomy (eLife, 2017). Instead of using authorship language CRediT refers to scholars as collaborators and provides a list of roles and responsibilities that individuals can use to describe each collaborators’ contribution (Atkins, 2016). This allows for increased transparency, credit, and accountability as it pertains to the scholarly work. Along a similar vein, a tiered authorship approach based on the type of contribution provided (principal staff, data contributor, statistical analysis, etc.) can be adopted. NICE: Crowd, for example, uses a tiered authorship approach. The requirements for authorship are made available to collaborators a priori and are available for others to refer to (https://osf.io/hmvny/).

As noted above, researcher and participant crowdsourced initiatives vary along several dimensions including: engagement, aims, accessibility, scope, and authorship. Researchers should take all of those dimensions into account when selecting an initiative to join. We aim to detail the utility of both researcher and participant crowdsourcing initiatives for cross-cultural research in the succeeding section.

Aim 3: Strengths and Challenges of Crowdsource Initiatives in Cross-Cultural Behavioral Research

Overview

In addition to the practical benefits of researcher and participant crowdsourced initiatives, scholars conceptualize several important scientific benefits of such research practices. These benefits include: replicability and generalizability, bias reduction, and increased team size. A commonly perceived challenge of engaging in crowdsourced initiatives, data quality, is also addressed.

Replications and generalizability

Researcher and participant crowdsourced initiatives can be used to conduct both direct and conceptual replications. Various types of replications and their associated benefits are discussed at length elsewhere (Schmidt, 2009; Simons, 2014; Uhlmann et al., 2019; Zwaan et al., 2018). Notable here is that researcher and participant crowdsourced initiatives can advance replications because adequate sample sizes can be recruited to power analyses, demographic diversity from multiple laboratories allows for the identification of the potential moderators, or idiosyncrasies within the original sample, and meta-analytic statistics can be calculated to examine the robustness of an effect.

Relatedly, research and participant crowdsourcing affords generalizability. Because researchers and participants can come from various locations,¹ there is heterogeneity in regards to demographic factors. If the findings are significant, the research team can be increasingly confident that the results are generalizable among groups with heterogeneous characteristics. For psychology specifically, better generalizability helps address criticisms of the field’s overreliance on American undergraduate students (Arnett, 2008) by diversifying sample composition beyond solely American students from a single institution. For the social sciences broadly speaking, this helps combat the use of WEIRD samples by including participants from various countries. If non-significant findings are found, labeled exploratory analyses can investigate why using demographic, regional, and cultural variables, and inspire future research examining potential moderators. While researcher and participant crowdsourcing initiatives aim to recruit large and diverse samples to increase the generalizability of results, and accuracy in effect size estimation, we recognize that while geographical diversity is often present, there is room for improvement in the representation of marginalized groups, especially people of color (Moshontz et al., 2018; Sierra-Mercado & Lázaro-Muñoz, 2018; Syed et al., 2011) and there are steps that all scientists can take to be more inclusive in their recruitment strategies (Chaudhary & Berhe, 2020; Yancey et al., 2006).

Researcher assumptions and biases

Researcher crowdsourcing has the ability to potentially reduce bias and increase accuracy/validity. Researchers study what they are interested in (Fiske, 2010) and all individuals bring their assumptions, prior experiences, and biases into the process (Cole & Stewart, 2001; Fisher et al., 2002; Fiske, 2010; Henrich et al., 2010). Researcher diversity is often an aim of many researcher crowdsourced initiatives (Cuccolo & Irgens, 2019; Moshontz et al., 2018; Swami et al., 2020) that requires effortful action (e.g., advertising the research opportunity, planning committees, outreach) to bring to fruition. The effortful inclusion of multiple researchers with varied demographics and diverse priors (e.g., ones’ assumptions, perspectives, experiences) facilitates communication about a project’s development, implementation, and interpretation (Moussa Rogers, personal communication, July 9, 2020). These multiple perspectives can serve to check assumptions and biases, and promote thoughtful consideration of explanations that best describe the data (Fiske, 2010; Silberzahn et al., 2018).

Team size benefits

An important benefit derived from researcher crowdsourcing is the impact of generated scholarly activity. Indeed, collaboration (i.e., co-authorship) is positively associated with citation rates, and number of institutions and countries involved has also been noted to exert a positive effect on article impact (Larivière et al., 2015; Wuchty et al., 2007). Collaboration has also been noted to have a positive relationship with an article’s impact, independently of the contribution’s novelty (Lee et al., 2015). Relatedly, open science practices used by many researcher crowdsourced initiatives (e.g., NICE: Crowd; PSA) have a beneficial impact on article metrics. For example, making data publicly accessible is related to an increase in citations independent of factors like date of publication and journal impact factor (Piwowar et al., 2007). Further still, some evidence suggests that sample size can influence article citations (Tahamtan et al., 2016). Given both researcher and participant crowdsourced initiatives can increase sample size all contributors may see a return on investment in terms of article metrics.

Data quality

It is important to note that previously mentioned benefits of sample size and power for statistical inferences are only relevant to the extent that the data accrued from participants is quality.

Policies and procedures for data quality in researcher crowdsourced initiatives

In regards to researcher crowdsourced initiatives, policies and procedures are usually in place to ensure data quality, such as thorough and clear data collection protocol videos (e.g., PSA) or documents (e.g., NICE: Crowd). Additional protocols such as the use of pre-registration templates and open data policies (e.g., NICE: Crowd) may be in place to ensure data quality, and allow for the checking of data if quality issues are suspected. Relatedly, another benefit of the aforementioned data quality policies and procedures, specifically open science initiatives, is increased accountability and transparency which can subsequently influence the public’s trust in science. Adopting such practices not only benefits researchers, but science as a whole, and the public’s perception of scientific findings (Vazire, 2017).

Online data quality

In regards to online participant crowdsourced initiatives, researchers may be wary of the quality of data generated given the rise of VPN or VPS use which allows for automated scripts generating random data (Kennedy et al., 2018). To combat this issue, researchers can run code written by Waggoner et al. (2019) to identify potentially problematic respondents who are likely using VPN or VPS. Researchers can also utilize reCAPTCHA to catch, and subsequently reject data from bots (Kennedy et al., 2018). Trust metrics can also be utilized if offered, or incorporated into projects, in order to help assess the validity of a particular user’s response (Alabri & Hunter, 2010; Peer et al., 2014). For a more thorough discussion on designing surveys to optimize data quality see: Couper et al. (2001), Hauser and Schwarz (2016), and Peer et al. (2014). Online participant crowdsourced initiatives can, however, also offer benefits to data collection in specific instances. For example, mTurk can reduce experimenter bias and social desirability given the lack of face-to-face contact, allow for the recruitment of hard-to-reach/specific populations, and some evidence suggests that mTurk has good response rates for multi-part studies and longitudinal research (Berinsky et al.,2012; Buhrmester et al., 2011; Shank, 2016).

General notes on ensuring data quality

Researchers should also be mindful that the quality of data reported by participants can vary as a function of demographic factors like age, experience with the subject matter, and familiarity with the protocol. It is necessary to be aware of these observer biases, and outline methods of handling data bias a priori (Dickinson et al., 2010). Additionally, some researchers may wish to take precautionary steps to circumvent these concerns, for example, by building practice sessions into the project protocol. Similarly, when recruiting data from a wide variety of persons, researchers should be aware of temporal and spatial heterogeneity in sampling efforts that could result in biased estimates. In regards to time, researchers may consider life and historical events affecting the sample, as well as the amount of time and effort the individual spends on task. Not specifying expectations and participation prerequisites can bias data, and as such researchers should make sure participants are informed about project requirements. Further, while more people may be reached through both researcher and participant crowdsourced initiatives it is important for researchers to note that not everyone will be reached, and sampling bias may still be present.

While Wutich and Brewis (2019) aim to provide recommendations for ensuring data quality for ethnographic research, we believe their recommendations are applicable to researcher crowdsourced initiatives as well. Recommendations include the development of a thorough data collection protocol that minimizes social desirability bias and simple language. Additionally, verbal and written data should be supplemented with observations, and all suspected data quality issues should be reported (e.g., participant was sick; participant appeared inattentive). Post-data collection researchers should check data for completeness, accuracy, currency, consistency, and adherence to the assumptions of planned statistical analyses (e.g., normality; Mertler & Reinhart, 2016).

Cross-Cultural Specific Crowdsourced Initiative

To our knowledge, there is only one organized, large-scale, on-going researcher crowd-sourcing initiative that focuses specifically on cross-cultural research; Psi Chi’s Network for International Collaborative Exchange (NICE). NICE is a program that aims to facilitate cross-cultural research among Psi Chi members and non-members both within, and outside of the United States. One specific component within the NICE is Crowd, a cross-cultural researcher crowdsourcing initiative where one to two research projects are selected for cross-cultural dissemination each year. In terms of structure, NICE has a core planning committee that selects a project to implement (via a crowdsourced submission process) and engages in frequent communication with contributors to convey protocol information, provide project assistance, and facilitate manuscript preparation.

The NICE: Crowd process operates as follows: early in the calendar year, the NICE committee sends out a call for project proposals to various organization listserv and Psi Chi chapters. Contributors are invited to submit a project proposal in the form of a completed pre-registration template for cross-cultural implementation at the start of the (US and European) academic year. Once they receive the proposals, the NICE planning committee evaluates them with a standardized rubric that focuses on the feasibility and sensitivity of the research for cross-cultural implementation. The committee chooses one or two projects which are subsequently advertised in order to garner “a crowd.” The NICE planning committee works with project authors to disseminate (and translate) measures and protocols, and provide IRB assistance to contributors running the project at their home institutions. Once a full academic year has elapsed, the planning committee works with project authors to facilitate manuscript preparation. Contributors can volunteer to help with various sections of the manuscript based on their interests and specialties. The collaborative nature of NICE allows for multiple viewpoints to be considered from research conception to dissemination of results. Notably, academics at various levels are invited to participate in NICE (i.e., undergraduate students, graduate students, full-time and part-time faculty).

NICE: Crowd may be particularly useful for psychologists who want to focus specifically on cross-cultural work, those with limited resources (as all materials are provided free of cost) or those interested in conducting a cross-cultural project who have a limited network (as the chosen project is disseminated to NICE collaborators across all countries with participating collaborators—seven countries in 2018), as well as faculty interested in increasing student interest in diversity, and those who want to conduct impactful research but have limited time (the burden of data collection is diffused across contributors). Another practical benefit of researcher crowdsourcing for cross-cultural work is the facilitation of research material translation into local languages (Uhlmann et al., 2019). This was done for the NICE: Crowd 2018 to 2019 project—collaboration between local researchers and members of the planning committee with translation experience allowed for several measures to be translated from English into German.

Given that researcher crowdsourced initiatives can be a means of improving behavioral science, it is crucial to understand the scientific community’s perceptions of these projects and to highlight for researchers the benefits and barriers which can inform future researcher crowdsourced project development. Despite the potential impact of such research, we have been unable to identify such perception studies in the literature. The remainder of this manuscript will describe contributor perceptions of participating a researcher crowdsourced initiative, the 2018 to 2019 NICE: Crowd project.

Aim 4: Contributor Perceptions from a Cross-Cultural Initiative: NICE: Crowd

To this end, we would like to highlight contributors’ perceptions of participating in a specific cross-cultural researcher crowdsourced initiative, NICE: Crowd. We integrate literature on the benefits of open science initiatives and cross-cultural research where applicable. Individuals who signed up to participate in NICE: Crowd 2018 to 2019 were contacted by the first author via an email detailing the aim of the current research and a link to a Qualtrics survey. The authors created the Qualtrics survey to assess perceived benefits, perceived strengths, general experience, and areas for improvement for NICE: Crowd. The question formats ranged from forced choice, multiple choice, rank order, and open ended. The protocol was approved by the first authors’ institutional review board. Out of 32 contributors contacted, N = 14 completed at least part of the survey. Of these 13 provided information about their current position; the majority were faculty members (N = 9; 69.2%), with the remaining portion being equally split between graduate students (n = 2, 15.4%) and undergraduate students (n = 2, 15.4%) who collaborated. Fifty percent of respondents indicated that their role in the project was that of “Research Assistant or Data Collector” while the other 50% indicated that they were the “Principal Investigator” for their location.

Benefits of the NICE: Crowd

Participants were presented with a list of benefits of participation in NICE: Crowd and could select all the benefits that applied. See Supplementary Materials for a full list of questions and response options. The frequency of responses for each item can be seen in Table 1.

Table 1.

Perceived Benefits of Participating in NICE: Crowd.

Benefit	% Endorsement
Exposure to open science Initiatives	78.6%
Exposure to Crowdsourcing	71.4%
A dataset that will generate scholarly activity	71.4%
Exposure to cross-cultural research	50.0%
Experience in authorship role	35.7%
Data to use for independent study/class project/senior thesis/etc.	35.7%
Networking opportunities	28.6%
Research experience	28.6%
Opportunities to mentor students in research methods	21.4%
Exposure to the scientific method	21.4%
Opportunities to mentor students in research methods	21.4%
Project Management Skills	21.4%
Experience in research assistant/data collector role	14.3%
Better training in the scientific method	7.1%
Experience in principle investigator role	7.1%

Collaborators responding to the survey were also asked to rank order from most to least important perceived benefits of Crowd. See Supplementary Material for all questions and response options. Learning open science principles and practices and having access to a diverse data set were most frequently ranked as the number one most important perceived benefit, less of a time burden collecting data was most frequently ranked as number two and Educating/Mentoring Students was most commonly ranked as number three. A full breakdown can be seen in Supplementary Table 2. When collaborators were asked to describe the strengths of NICE: Crowd in an open-ended format, responses centered around NICE: Crowd as an introduction to open science and crowdsourcing. Other responses focused on the methodological rigor, collaboration, and cross-cultural focus.

Other Characteristics

Out of the 12 respondents who indicated if they had previously participated in other researcher crowdsourced projects, half noted (N = 6), this was their first time participating in a researcher crowdsourced project. Of those who had previously participated in a collaborative project (N = 6), 12.5% indicated they had a significantly more positive experience with NICE: Crowd compared to other projects, 50.0% indicated they had a slightly more positive experience with NICE: Crowd compared to other projects, and 37.5% indicated the experiences were roughly equivalent. Participants were asked to provide open ended responses regarding what made their experience with NICE: Crowd more positive than their experience with other projects. Clear participation guidelines, expectations, and frequent communication were most common elements of responses regarding the positive experiences with NICE: Crowd. Along a similar vein, contributors were asked to describe the ways in which NICE: Crowd could be improved. The suggestions focused on two main themes: providing more guidance about the Open Science Framework and increasing communication between collaborators (as opposed to communication between the planning committee and contributors). These results indicate that researchers are interested in learning more about and using open science initiatives and therefore, future initiatives should take care to properly facilitate instruction on how to use such platforms. It also indicates that researchers are interested in building relationships with their collaborators, but such increased communication may be conceptualized differently than networking, which ranked lower than most other benefits.

Implication of the Benefits of NICE: Crowd

The open science movement in psychology has been steadily gaining attention and momentum. The use of open science practices in NICE: Crowd allows researcher collaborators to become familiar with open science initiatives and tools while concurrently promoting the best methodological practices. For example, contributors gained experience with data and measure repositories (i.e., the Open Science Framework or OSF). OSF repositories provide researchers with an electronic space to store their pre-registrations, protocols, data, and even pilot studies. Making such materials (e.g., protocols) freely available to other researchers provides several benefits to psychological science such as transparency and quality control. Independent researchers are able to see how a particular study was conducted and the analyses used to arrive at particular results, then use them to conduct their own replications. Being able to easily and accurately conduct replications or to further investigate particular effects helps flush out false positives, false negatives, and potentially even fraud (Schmidt, 2009).

Speaking to the benefits of NICE: Crowd’s use of open science initiatives, NICE: Crowd contributors wishing to submit a project proposal for implementation must do so by completing a pre-registration template. Pre-registration templates are open science initiatives that afford several benefits to data quality. Pre-registrations are a priori commitments to research plans where researchers detail the sample, measures, materials, research questions, hypotheses, and analyses that will be used in a given study. This focus on study design allows for methodological rigor and ability to evaluate both the study design and planned analyses to test the hypothesis, instead of placing the focus solely on the study’s outcome (significance). These a priori plans also enhance the validity of research by helping to ensure methodological soundness and fidelity to research designs (Nosek & Lakens, 2014). Specifying analyses ahead of time helps protect researchers against the use of poor statistical techniques such as p-hacking. Pre-registrations may be particularly useful for circumventing HARKing (hypothesizing after results are known; Kerr, 1998). It has been noted that individuals are likely to interpret information as being in line with what they want to be true, or what they want to find (Bastardi et al., 2011); pre-registrations can thus protect researchers from their own cognitive biases in which they are motivated to generate narrative desired by themselves or journal editors. Ultimately, pre-registrations increase confidence in both positive and negative results (van’t Veer & Giner-Sorolla, 2016).

Increasing the accessibility of measures and data through their availability on repositories helps facilitate research among diverse groups of researchers. Because data and measure repositories are freely available, individuals’ financial status and/or institutional endowment do not present barriers to accessing and utilizing knowledge. Given that women and ethnic/racial minorities are underrepresented in science, and individuals are often limited by the resources of their institution, having materials, measures, and data freely available make it easier for individuals to conduct replications, access knowledge, and be part of the scientific community (e.g., comment on current research, seek collaboration; Alzahrani, 2011; Grahe et al., 2019; Santo et al., 2009). Indeed, the benefits of access to a diverse dataset and reducing the time burden of data collection speak to this point.

Reduction in barriers to data accessibility can increase the ease with which faculty with minoritized identities—who typically have more of a teaching load/departmental service—participate in research. Likewise, publication bias in which significant findings are more likely to be published, have skewed the understanding of various relationships and effect sizes (Anderson & Maxwell, 2017; Munafò et al., 2017). Indeed, replication of previously published effects has been deemed an issue in science (Ioannidis, 2005). Disseminating findings from pilot studies or studies with negative (non-significant) findings through the use of repositories, pre-registrations, and registered reports can help provide more accurate estimates of effect sizes (Munafò et al., 2017).

Another perceived benefit of NICE: Crowd is the educating and mentoring of students. As open science initiatives become more commonplace, it is critical for current students to be comfortable using such methodology. NICE: Crowd provides an avenue where students can gain hands on experience with various open science initiatives (mentioned above). Furthermore, NICE: Crowd provides faculty with a way to expose a large number of students to research, as the projects lend themselves to implementation in the classroom. Professors have used NICE: Crowd in the classroom to assist in teaching research methods, statistics, honors level forums, and/or honors thesis, and senior projects. Along a similar vein, NICE: Crowd provides students with hands on research experience where they can make a meaningful contribution to the field through data collection and potentially earn authorship. Indeed, collaborators are eligible for authorship if certain criteria are met. The 2018 to 2019 NICE: Crowd manuscript is currently being written by a diverse team of collaborators, all of whom are using their unique skill sets to advance the manuscript quality. This includes several students who will be authors on the final manuscript. Authorship for this NICE: Crowd 2018 to 2019 manuscript is divided into tiers based on contributions, and within each tier the order of authors will be alphabetical. Authorship criteria are available to contributors a priori: https://osf.io/hmvny/. Collaborators are able to choose which section of the manuscript they would like to write given their strengths; for example, one group is focusing on specific aspects of the statistical analyses.

Conclusion

In closing, researcher and participant crowdsourcing can advance cross-cultural psychology through several avenues: increasing generalizability and reliability, decreasing time burdens, democratizing research, educating individuals on open science, and providing mentorship. Generalizability and reliability are advanced through increased power and sample diversity, as well as through decreased ethnocentric bias as a result of having researchers with varying demographic backgrounds on the research team. Time burdens are decreased as a result of data pooling—each researcher need only to recruit a fraction of the sample. Additionally, within researcher crowdsourced initiatives the provision of measures and protocols save researchers time during the research planning process. Researcher and participant crowdsourcing also helps to democratize research by making science available to more people. For research crowdsourced initiatives the provision of resources, measures, protocols, and guidance (free of cost) enables individuals who are restricted in their resources to engage in high quality, publishable research. Some researcher crowdsourcing initiatives focus heavily on open science (e.g., NICE: Crowd), and as such, may be great learning tools for those wishing to gain a better perspective on the open science movement. Finally, researcher crowdsourcing provides an opportunity for researchers to be mentored, and also mentor others. Researchers can bring their students on and assist them throughout the research process as they make meaningful contributions to cross-cultural behavioral science. Concurrently, researchers can learn from other contributors and project organizers about open science, the phenomenon under study, and more. Researcher and participant crowd-sourcing offers significant benefits to cross-cultural researchers who wish to further develop their research repertoire.

Supplemental Material

SCCR_Supplementary_materials_7_13_20 – Supplemental material for What Crowdsourcing Can Offer to Cross-Cultural Psychological Science

Supplemental material, SCCR_Supplementary_materials_7_13_20 for What Crowdsourcing Can Offer to Cross-Cultural Psychological Science by Kelly Cuccolo, Megan S. Irgens, Martha S. Zlokovich, Jon Grahe and John E. Edlund in Cross-Cultural Research

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Supplemental Material

Supplemental material for this article is available online.

Notes

Author Biographies

Kelly Cuccolo, is a doctoral candidate and lecturer at the University of North Dakota. Her main research interests lie in understanding the effects of various dietary and physical activity habits on psychological well-being, with a specific focus on understanding the influence of objectification on women’s decisions to engage in physical activity. She is additionally interested in meta-science, cross-cultural crowdsourcing initiatives, and the teaching of psychology. She served as the Chair of Psi Chi’s Network for International Collaborative Exchange (NICE) from 2017-2019, and is currently still actively participating in the initiative as a contributor.

Megan S. Irgens, MA, is a doctoral candidate in the Clinical Psychology Department at the University of Arizona. Her research interests broadly concern the intersection of psychology and law. More specifically, she studies family cohesion, how families communicate, and the shifting of roles and responsibilities of family members when a family member is involved in the legal system. Megan also serves as the past Chair of Psi Chi’s Network for International Collaborative Exchange.

John E. Edlund, is an Associate Professor of Psychology at the Rochester Institute of Technology. He serves as the Research Director for Psi Chi and as an associate editor at multiple academic journals. He recently published an edited text entitled “Advanced Research Methods for the Social and Behavioral Sciences.” His research focuses are on meta-science (crosstalk, consent, replicability) and on human mating behaviors (explored from the perspectives of evolutionary and social psychology).

Martha S. Zlokovich, PhD, is a Developmental Psychologist and Executive Director of Psi Chi, the International Honor Society in Psychology. She was recently elected to a second term as Division 52 Treasurer, is Past President of the Society for Psychologists in Management (SPIM), and has served as joint conference chair with Division 13. Previously she was a Professor of Psychology at Southeast Missouri State University, Psychology Department Chair, Interim Chair of the Center for Scholarship in Teaching and Learning, and Faculty Senate Chair. She was elected to the Cape Girardeau Public Schools Board, Psi Chi Board President, and President of the Association of College Honor Societies Board. Her research interests are international crowd-sourced research methods, young adult sexuality and contraception, and student study habits, study beliefs, and persistence to graduation.

Jon Grahe, is a Professor of Psychology and Department Chair at Pacific Lutheran University and the Managing Executive Editor of The Journal of Social Psychology with a previous leadership role as President of Psi Chi, the International Honor Society for Psychology. He was recognized as Fellow for both Western Psychological Association and Midwestern Psychological Association. Though trained as a social psychologist, his scholarships focuses on designing and administering large scale collaborative research projects for undergraduates including, among others, the Collaborative Replications and Education Project (CREP) and the Emerging Adulthood Measured at Multiple Institutions (EAMMi2) project.

References

Alabri

Hunter

(2010). Enhancing the quality and trust of citizen science data. In 2010 IEEE sixth international conference on e-science (pp. 81–88). IEEE.

Alzahrani

J. A.

(2011). Overcoming barriers to improve research productivity in Saudi Arabia. International Journal of Business and Social Science, 2(19), 50–57.

Anderson

S. F.

Maxwell

S. E.

(2017). Addressing the “replication crisis”: Using original studies to design replication studies with appropriate statistical power. Multivariate Behavioral Research, 52(3), 305–324. https://doi.org/10.1080/00273171.2017.1289361

Arnett

J. J.

(2008). The neglected 95%: why American psychology needs to become less American. American Psychologist, 63(7), 602–614. https://doi.org/10.1037/14805-008

Arza

Fressoli

(2017). Systematizing benefits of open science practices. Information Services & Use, 37(4), 463–474. https://doi.org/10.3233/ISU-170861

Asendorpf

J. B.

Conner

De Fruyt

De Houwer

Denissen

J. J.

Fiedler

Funder

D. C.

Kliegl

Nosek

B. A.

Perugini

(2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27(2), 108–119. https://doi.org/10.1002/per.1919

Atkins

(2016). Author credit: PLOS and CRediT update. PLOS. https://theplosblog.plos.org/2016/07/author-credit-plos-and-credit-update/

Bastardi

Uhlmann

E. L.

Ross

(2011). Wishful thinking: Belief, desire, and the motivated evaluation of scientific evidence. Psychological Science, 22(6), 731–732. https://doi.org/10.1177/0956797611406447

Berinsky

A. J.

Huber

G. A.

Lenz

G. S.

(2012). Evaluating online labor markets for Experimental research: Amazon.com’s mechanical Turk. Political Analysis, 20(3), 351–368. https://doi.org/10.1093/pan/mpr057

10.

BrightFocus Foundation. (2018, May 7). First-Ever Citizen Science Project for Alzheimer’s Research Gets 10,000th Volunteer. https://www.brightfocus.org/alzheimers/news/first-ever-citizen-science-project-alzheimers-research-gets-10000th-volunteer

11.

Buhrmester

Kwang

Gosling

S. D.

(2011). Amazon’s mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5. https://doi.org/10.1037/14805-009

12.

Chaudhary

Berhe

A. A.

(2020). Ten simple rules for building an anti-racist lab. EcoEvoRxiv. https://doi.org/10.32942/osf.io/4a9p8

13.

Cole

E. R.

Stewart

A. J.

(2001). Invidious comparisons: Imagining a psychology of race and gender beyond differences. Political Psychology, 22(2), 293–308. https://doi.org/10.1111/0162-895X.00240

14.

Couper

M. P.

Traugott

M. W.

Lamias

M. J.

(2001). Web survey design and administration. Public Opinion Quarterly, 65(2), 230–253. https://doi.org/10.1086/322199

15.

Cuccolo

Irgens

M. S.

(2019). OSF: NICE Crowd call for proposals [Webinar]. Psi Chi. https://www.psichi.org/page/Webinar06272019#.XwdsAyhKhPZ

16.

Del Savio

Prainsack

Buyx

(2016). Crowdsourcing the Human Gut. Is crowdsourcing also ‘citizen science’? Journal of Science Communication, 15(3), 1–16.

17.

Dickinson

J. L.

Zuckerberg

Bonter

D. N.

(2010). Citizen science as an ecological research tool: Challenges and benefits. Annual Review of Ecology, Evolution, and Systematics, 41(1), 149–172.

18.

Ebersole

C. R.

Atherton

O. E.

Belanger

A. L.

Skulborstad

H. M.

Allen

J. M.

Banks

J. B.

Baranski

Bernstein

M. J.

Bonfiglio

D. B.

Boucher

Brown

E. R.

(2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68–82. https://doi.org/10.1016/j.jesp.2015.10.012

19.

eLife. (2017). Enabling the contributor roles taxonomy for author contributions. Retrieved March 24, 2020, from https://elifesciences.org/inside-elife/f39cfcf5/enabling-the-contributor-roles-taxonomy-for-author-contributions

20.

Eitzel

M. V.

Cappadonna

J. L.

Santos-Lang

Duerr

R. E.

Virapongse

West

S. E.

Kyba

Bowser

Cooper

C. B.

Sforzi

Metcalfe

A. N.

(2017). Citizen science terminology matters: Exploring key terms. Citizen Science: Theory and Practice, 2(1), 2. http://doi.org/10.5334/cstp.96

21.

Fisher

C. B.

Hoagwood

Boyce

Duster

Frank

D. A.

Grisso

Levine

R. J.

Macklin

Spencer

M. B.

Takanishi

Trimble

J. E.

(2002). Research ethics for mental health science involving ethnic minority children and youths. American Psychologist, 57(12), 1024. https://doi.org/10.1037/0003-066X.57.12.1024

22.

Fiske

S. T.

(2010). Venus and Mars or down to Earth: Stereotypes and realities of gender differences. Perspectives on Psychological Science, 5(6), 688–692. https://doi.org/10.1177/1745691610388768

23.

Gelfand

M. J.

Raver

J. L.

Nishii

Leslie

L. M.

Lun

Lim

B. C.

Duan

Almaliach

Ang

Arnadottir

Aycan

(2011). Differences between tight and loose cultures: A 33-nation study. Science, 332(6033), 1100–1104. https://doi.org/10.1126/science.1197754

24.

Grahe

J. E.

Brandt

IJzerman

Cohoon

(2014). Replication education. APS Observer, 27(3). https://www.psychologicalscience.org/observer/replication-education

25.

Grahe

J. E.

Cuccolo

Leighton

D. C.

Cramblet Alvarez

L. D.

(2020). Open science promotes diverse, just, and sustainable research and educational outcomes. Psychology Learning & Teaching, 19(1), 5–20.

26.

Handler

R. A.

Conill

R. F.

(2016). Open data, crowdsourcing and game mechanics. A case study on civic participation in the digital age. Computer Supported Cooperative Work (CSCW), 25(2–3), 153–166. https://doi.org/10.1007/s10606-016-9250-0

27.

Harms

P. D.

DeSimone

J. A.

(2015). Caution! MTurk workers ahead—Fines doubled. Industrial and Organizational Psychology, 8(2), 183–190. https://doi.org/10.1017/iop.2015.23

28.

Hauser

D. J.

Paolacci

Chandler

J. J.

(2018). Common Concerns with MTurk as a Participant Pool: Evidence and solutions. In Kardes

Herr

Schwarz

(Eds), Handbook in Research Methods in Consumer Psychology. (1^st Ed., 319-337). Routledge. https://doi.org/10.31234/osf.io/uq45c

29.

Hauser

D. J.

Schwarz

(2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48(1), 400–407. https://doi.org/10.3758/s13428-015-0578-z

30.

Henrich

Heine

S. J.

Norenzayan

(2010). Most people are not WEIRD. Nature, 466(7302), 29. https://doi.org/10.1038/466029a

31.

Ioannidis

J. P.

(2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124

32.

Jameson

D. A.

(1994). Strategies for overcoming barriers inherent in cross-cultural research. Bulletin of the Association for Business Communication, 57(3), 39–41.

33.

John

L. K.

Loewenstein

Prelec

(2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953

34.

Kam

C. D.

Wilking

J. R.

Zechmeister

E. J.

(2007). Beyond the “narrow data base”: Another convenience sample for experimental research. Political Behavior, 29(4), 415–440. https://doi.org/10.1007/s11109-007-9037-6

35.

Kennedy

Clifford

Burleigh

Jewell

Waggoner

(2018). The shape of and solutions to the MTurk quality crisis. Available at SSRN 3272468.

36.

Kerr

N. L.

(1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4

37.

Kidwell

M. C.

Lazarević

L. B.

Baranski

Hardwicke

T. E.

Piechowski

Falkenberg

L. S.

Kennett

Slowik

Sonnleitner

Hess-Holden

Errington

T. M.

(2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biology, 14(5), e1002456. http://doi.org/10.1371/journal.pbio.1002456

38.

Klein

Ratliff

Vianello

Adams

Jr. Bahník

Bernstein

Bocian

Brandt

Brooks

Brumbaugh

Cemalcilar

(2014). Data from investigating variation in replicability: A “many labs” replication project. Journal of Open Psychology Data, 2(1), e4. http://doi.org/10.5334/jopd.ad

39.

Klein

R. A.

Vianello

Hasselman

Adams

B. G.

Adams

R. B.

Jr. Alper

Aveyard

Axt

J. R.

Babalola

M. T.

Bahník

Š.

Batra

(2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. https://doi.org/10.1177/2515245918810225

40.

Larivière

Gingras

Sugimoto

C. R.

Tsou

(2015). Team size matters: Collaboration and scientific impact since 1900. Journal of the Association for Information Science and Technology, 66(7), 1323–1332. https://doi.org/10.1002/asi.23266

41.

Laursen

Hunter

A. B.

Seymour

Thiry

Melton

(2010). Undergraduate research in the sciences: Engaging Students in Real Science. John Wiley & Sons.

42.

Lee

Y. J.

Arida

J. A.

Donovan

H. S.

(2017). The application of crowdsourcing approaches to cancer research: A systematic review. Cancer Medicine, 6(11), 2595–2605. https://doi.org/10.1002/cam4.1165

43.

Lee

Y. N.

Walsh

J. P.

Wang

(2015). Creativity in scientific teams: Unpacking novelty and impact. Research Policy, 44(3), 684–697. https://doi.org/10.1016/j.respol.2014.10.007

44.

McKiernan

E. C.

Bourne

P. E.

Brown

C. T.

Buck

Kenall

Lin

Yarkoni

(2016). How open science helps researchers succeed. Elife, 5, e16800. https://doi.org/10.7554/elife.16800

45.

Mertler

C. A.

Reinhart

R. V.

(2016). Advanced and multivariate statistical methods: Practical application and interpretation. Taylor & Francis.

46.

Merton

R. K.

(1968). The Matthew effect in science. Science, 159(3810), 56–63. https://doi.org/10.1126/science.159.3810.56

47.

Miller

J. D.

Crowe

Weiss

Maples-Keller

J. L.

Lynam

D. R.

(2017). Using online, crowdsourcing platforms for data collection in personality disorder research: The example of Amazon’s Mechanical Turk. Personality Disorders: Theory, Research, and Treatment, 8(1), 26–34. https://doi.org/10.1037/per0000191

48.

Moshontz

Campbell

Ebersole

C. R.

IJzerman

Urry

H. L.

Forscher

P. S.

Grahe

J. E.

McCarthy

R. J.

Musser

E. D.

Antfolk

Castille

C. M.

Evans

T. R.

Fiedler

Flake

J. K.

Forero

D. A.

Janssen

S. M. J.

Keene

J. R.

Protzko

Aczel

. . . Chartier

C. R

(2018). The psychological science accelerator: Advancing psychology through a distributed collaborative network. Advances in Methods and Practices in Psychological Science, 1(4), 501–515. https://doi.org/10.1177/2515245918797607

49.

Munafò

M. R.

Nosek

B. A.

Bishop

D. V.

Button

K. S.

Chambers

C. D.

Du Sert

N. P.

. . . Ioannidis

J. P.

(2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 1–9. https://doi.org/10.1038/s41562-016-0021

50.

Nosek

B. A.

Ebersole

C. R.

DeHaven

A. C.

Mellor

D. T.

(2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114

51.

Nosek

B. A.

Lakens

(2014). A method to increase the credibility of published results. Social Psychology, 45(3), 137–141. http://doi.org/10.1027/1864-9335/a000192.

52.

Palmer

J. C.

Strickland

(2016). A beginner’s guide to crowdsourcing Strengths, limitations and best practices for psychological research. Retrieved December 2, 2019, from https://www.apa.org/science/about/psa/2016/06/changing-minds

53.

Peer

Vosgerau

Acquisti

(2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46(4), 1023–1031. https://doi.org/10.3758/s13428-013-0434-y

54.

Piwowar

H. A.

Day

R. S.

Fridsma

D. B.

(2007). Sharing detailed research data is associated with increased citation rate. PLoS One, 2(3), e308. https://doi.org/10.1371/journal.pone.0000308

55.

Santo

S. A.

Engstrom

M. E.

Reetz

Schweinle

W. E.

Reed

(2009). Faculty productivity barriers and supports at a school of education. Innovative Higher Education, 34(2), 117–129. https://doi.org/10.1007/s10755-009-9098-z

56.

Schmidt

(2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2), 90–100. https://doi.org/10.1037/a0015108

57.

Schweinsberg

Madan

Vianello

Sommer

S. A.

Jordan

Tierney

Awtrey

Zhu

L. L.

Diermeier

Heinze

J. E.

Srinivasan

(2016). The pipeline project: Pre-publication independent replications of a single laboratory’s research pipeline. Journal of Experimental Social Psychology, 66, 55–67. https://doi.org/10.1016/j.jesp.2015.10.001

58.

Shank

D. B.

(2016). Using crowdsourcing websites for sociological research: The case of Amazon Mechanical Turk. The American Sociologist, 47(1), 47–55. https://doi.org/10.1007/s12108-015-9266-9

59.

Sierra-Mercado

Lázaro-Muñoz

(2018). Enhance diversity among researchers to promote participant trust in precision medicine research. The American Journal of Bioethics, 18(4), 44–46. https://doi.org/10.1080/15265161.2018.1431323

60.

Silberzahn

Uhlmann

E. L.

Martin

D. P.

Anselmi

Aust

Awtrey

Bahník

Š.

Bai

Bannard

Bonnier

Carlsson

(2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. https://doi.org/10.1177/2515245917747646

61.

Simons

D. J.

(2014). The value of direct replication. Perspectives on Psychological Science, 9(1), 76–80. https://doi.org/10.1177/1745691613514755

62.

Swami

Tran

U. S.

Barron

Afhami

Aimé

Almenara

C. A.

. . . Argyrides

(2020). The Breast Size Satisfaction Survey (BSSS): Breast size dissatisfaction and its antecedents and outcomes in women from 40 nations. Body Image, 32, 199–217. https://doi.org/10.1016/j.bodyim.2020.01.006

63.

Syed

Azmitia

Cooper

C. R.

(2011). Identity and academic success among underrepresented ethnic minorities: An interdisciplinary review and integration. Journal of Social Issues, 67(3), 442–468. https://doi.org/10.1111/j.1540-4560.2011.01709.x

64.

Tahamtan

Afshar

A. S.

Ahamdzadeh

(2016). Factors affecting number of citations: a comprehensive review of the literature. Scientometrics, 107(3), 1195–1225. https://doi.org/10.1007/s11192-016-1889-2.

65.

Uhlmann

E. L.

Ebersole

C. R.

Chartier

C. R.

Errington

T. M.

Kidwell

M. C.

Lai

C. K.

McCarthy

R. J.

Riegelman

Silberzahn

Nosek

B. A.

(2019). Scientific Utopia III: Crowdsourcing science. Perspectives on Psychological Science, 14(5), 711–733. https://doi.org/10.1177/17456916198505

66.

van’t Veer

A. E.

Giner-Sorolla

. (2016). Pre-registration in social psychology—A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12. https://doi.org/10.1016/j.jesp.2016.03.004

67.

Vazire

(2017). Quality uncertainty Erodes trust in science. Collabra: Psychology, 3(1), 1.

68.

Wagge

Baciu

Banas

Nadler

J. T.

Schwarz

Weisberg

IJzerman

Legate

Grahe

(2018). A Demonstration of the collaborative replication and education project: Replication attempts of the red-romance effect. Collabra: Psychology, 5(1), 5. https://doi.org/10.1525/collabra.177

69.

Waggoner

Kennedy

Clifford

(2019). Detecting fraud in online surveys by tracing, scoring, and visualizing IP addresses. Journal of Open Source Software, 4(37), 1285. https://doi.org/10.21105/joss.01285

70.

Whitla

(2009). Crowdsourcing and its application in marketing activities. Contemporary Management Research, 5(1), 15–28. https://doi.org/10.7903/cmr.1145

71.

Wiggins

Crowston

(2011). From conservation to crowdsourcing: A typology of citizen science. In 2011 44th Hawaii international conference on system sciences (pp. 1–10). IEEE.

72.

Wuchty

Jones

B. F.

Uzzi

(2007). The increasing dominance of teams in production of knowledge. Science, 316(5827), 1036–1039. https://doi.org/10.1126/science.1136099

73.

Wutich

Brewis

(2019). Data collection in cross-cultural ethnographic research. Field Methods, 31(2), 181–189. https://doi.org/10.1177/1525822X19837397

74.

Yancey

A. K.

Ortega

A. N.

Kumanyika

S. K.

(2006). Effective recruitment and retention of minority research participants. Annual Review of Public Health, 27(1), 1–28. https://doi.org/10.1146/annurev.publhealth.27.021405.102113

75.

Zwaan

R. A.

Etz

Lucas

R. E.

Donnellan

M. B.

(2018). Making replication mainstream. Behavioral and Brain Sciences, 41, E120. https://doi.org/10.1017/S0140525X17001972

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.22 MB