Increasing research impact with citizen science: The influence of recruitment strategies on sample diversity

1. Introduction

In the European Union (EU)-wide efforts to create more societal value from scientific knowledge, also known as valorisation, the participation of a diverse range of actors in the production of knowledge is considered crucial. Besides the collaboration with businesses, public authorities and other stakeholders, increasing attention is being paid to the interaction with the broader public in the form of citizen science (CS). Following a steep growth of CS projects in number, scale and scope (Bonney et al., 2016; McKinley et al., 2017), influential international institutions have started advocating the idea of public participation in research as a source of inspiration for engaging and empowering citizens and a mechanism to improve science-society-policy interactions (European-Commission, 2016).

The promising discourse surrounding CS, and the generally recognized importance of public participation in research, are of considerable importance. The idea of responsible research and innovation (RRI), including public engagement, is defined as a `cross-cutting issue’ in the EU Research and Innovation funding programme Horizon 2020 (European-Commission, 2016). An independent High Level Group on maximizing the impact of EU Research & Innovation programmes recently identified CS as one of the key actions to strengthen the impact of European research and innovation (Lamy Committee, 2017). The current proposal for the next European Framework Programme suggests that the attention to citizen engagement in research is going to increase. ¹

The number of studies on the value, potential and possibilities of CS is rapidly expanding. However, to date, empirical studies on engaging a broad diversity of citizens in practice are scant. As we will show in section 3, a representative sample of volunteers is key for CS to fulfil its promises in terms of valorisation. The aim of this article is to twofold: (1) to provide more insight into the value, opportunities and means of engaging audiences that typically are not engaged with science; and (2) to explore the effect of inviting specific households to participate in CS projects, instead of using a generic invitation to the general public. Based on a rich dataset of five CS projects, our article analyses the influence of a targeted recruitment strategy versus a generic recruitment strategy on the sample of participants in CS projects, explores how the activity of participants relates to personal characteristics and looks at the question of how personal characteristics relate to impacts on confidence and awareness.

2. Rationales for CS

Science policies across the world have shown a growing interest in increasing the societal value from science over the past few decades. In relation to changing public and political expectations of science, policies of governments and research organizations have shifted their attention from concerns about quality and relevance to a focus on valorisation (Hessels et al., 2009), that is, to ‘the process of creating value from knowledge by making knowledge suitable and/or available for economic and/or societal use and translating that knowledge into products, services, processes and entrepreneurial activity’ (De Jong et al., 2015). One driver for this focus is a pressure for accountability, which implies the need to show that public investments in science lead to economic competitiveness, improved public policies or other forms of ‘social impacts’. Valorisation policy interventions include challenge-driven research programmes (Bos et al., 2014), the inclusion of impact assessment in university performance evaluations (Smith et al., 2011) and the stimulation of public engagement with science (Stilgoe et al., 2014). This article deals with the promotion of public engagement, particularly CS, as a strategy to stimulate valorisation, and accordingly does not focus on grassroots CS.

Even if CS projects generally have overlapping goals, on a higher level of abstraction the rationales for CS cited in the literature can roughly be classified as either substantive, instrumental or normative. The substantive rationale relates to the idea that CS might improve the quality of research or the final product. Public participation in research might not only raise new research questions that the scientific community had not yet considered, and generate more information on the research topic itself, but it might also suggest additional alternative or more specific solutions (Reed, 2008). The instrumental rationale for CS is rooted in the idea that the process of participation will increase the legitimacy and/or quality of the final product. Cooperation between scientists and the public is believed to reinforce mutual trust (Bäckstrand, 2003; Chandler et al., 2012). Another part of the instrumental rationale rests on the potential of citizen scientists to make large contributions to the quantity of data available to research professionals (Bonney et al., 2016; Cohn, 2008; Lidskog, 2008). The normative rationale for CS postulates that there is intrinsic value to and strong moral ‘ought’ for participation (Stilgoe et al., 2014). Indeed, scholars such as Buytaert et al. (2014) and Haywood (2016) argue that the engagement of the public in research processes makes science more democratic and ‘legitimate’, regardless of its effects on trust or the implementation of research results discussed in relation to the instrumental rationale.

We would argue that both the substantive and normative rationale for CS are almost by definition associated with knowledge valorisation. Projects initiated because of these rationales aim to contribute to the social impacts of science. As for the instrumental rationale, there can be various manifestations which are not necessarily linked with valorisation. Whereas projects that engage with citizens to increase the legitimacy of results typically aim for valorisation, research projects that invite citizens to participate in the collection of data from an efficiency perspective, will not always be oriented at generating social impacts or improving science literacy. In light of the work of Bonney et al. (2009), who made a classification of different CS projects, these will be primarily ‘contributory’ projects in which citizens merely collect data for scientists to use in their research.

3. Causal patterns between CS and the social impacts of science

Many CS projects are driven by a combination of substantive, instrumental and normative rationales. We would argue that, in principle, a project driven by any of the rationales can contribute to valorisation. When looking in more detail at the valorisation promises in relation to CS in the literature, one can distinguish at least seven different causal patterns assumed between CS and the social impacts of science:

A critical look at these claims reveals that they rest on a couple of crucial assumptions, which often remain rather implicit, both in policy documents and in scientific papers. In particular, the causal patterns 2, 3, 4, 6 and 7 all depend on the condition that a diverse (representative) sample of citizens participate in the research. If the (active) contributions of volunteers come mainly from a highly skewed sample of citizens, these promises will not be fulfilled. A second crucial condition to be noted is that the citizen scientists are affected by their participation in the project, in the sense that it changes or enriches their relationship with the object of study or with science in general. Only if this is the case, can CS enhance the public acceptance of scientific outcomes or increase the scientific literacy of participants.

We would argue that many CS projects do not fulfil these conditions. When looking at the educational background of CS participants, we find again and again that people with higher education are overrepresented in CS (Strasser and Haklay, 2018). Indeed, CS projects with over two-thirds of the participants holding a university degree are by no means exceptional (Brossard et al., 2005; Overdevest et al., 2004; Raddick et al., 2013). Furthermore, the empirical evidence with regard to improved understanding of the scientific process remains inconclusive at best, nor does participation necessarily result in, for instance, the adoption of pro-environmental attitudes. Improvements in content knowledge, however, seem to be better supported (Bonney et al., 2016; Brossard et al., 2005; Crall et al., 2013; Cronje et al., 2011; Jollymore et al., 2017; Strasser and Haklay, 2018).

The fact that many CS projects do not to fulfil these conditions may relate to the fact that not all projects are intended to improve the social impacts of science. Many projects, for instance, are intentionally designed to primarily increase the quantity of data or to answer specific scientific questions. We concur with Bonney et al. (2009) that such projects should not be held to standards for broadening the impact of science that they never intended to achieve. However, an alternative plausible explanation may relate to a lack of expertise or knowledge on how these conditions translate in design principles. Although recruiting participants is integral to the success of each individual CS project, very little has been published about different CS recruitment strategies. An exception is the paper by West and Pateman (2016), who, to this end, draw on the volunteering literature. They distinguish between three recruitment strategies: (1) word-of-mouth; (2) the use of third party organizations, such as volunteering agencies; and (3) the ‘scattergun approach’, that is, advertising to large numbers of people using, for instance, mass and social media. Whereas the first approach is likely to attract people who are already engaged in volunteering, the use of third party organizations may well function to reach specific groups (Ockenden, 2007; Unell and Castle, 2012). The ‘scattergun approach’ can be expected to create a bias towards people that are receptive to the particular media employed.

From a diversity perspective, a tension exists between the need to establish a legitimate cross-section of the population and the need to acknowledge and account for salient differences between groups (Blue et al., 2012). In this article, we focus on diversity in the sense of a representative sample of the population. In this light, this article presents a fourth strategy to recruit participants, which concerns sending out personal invitations to a random sample of the population. We refer to this strategy as a targeted strategy, as opposed to a non-targeted or generic invitation strategy, aligning well with the scattergun approach (West and Pateman, 2016) directed at all citizens of a specific area, but no one in particular. In section 5, we elaborate on the influence of recruitment strategy on the sample of participants and their motivation, followed by section 6 which evaluates the value of diversity from a research impact perspective. In the next section, we first explain how we conducted our research.

4. Research design

Our empirical research is based on five unique CS projects in the domain of water research in the Netherlands conducted in the period between 2016 and 2017, in cooperation with four different (drinking) water companies and over 1000 volunteers: the ‘Freshness of Water’ (2016), the ‘Clean Water Experiment’ (2017), ‘Citizen Science and Lime’ (2017), ‘Citizen Science and Lead’ (2017) and ‘Citizen Science and Hardness’ (2017–2018); see Table 1. The selection of these projects was guided by two considerations: (1) project relevance and (2) the availability of data. First, the selection of projects covers all CS projects in the drinking water sector in the Netherlands, as well as the country’s most comprehensive CS project related to surface water quality. Second, for all projects we have detailed empirical knowledge and data concerning the actual process. ² All five projects can be characterized as primarily contributory CS projects, in which members of the public merely collect data for scientists to use in their research (Bonney et al., 2009). Two of the projects used a generic recruitment strategy, while the other three followed a targeted recruitment strategy by personally contacting a random sample of specific households, using the client database of the cooperating water company.

OPEN IN VIEWER

Table 1.

Overview of the recruitment strategies, number of invitations, participants and response rate.

	Recruitment strategy	Number of invitations	Response rate (positive completes)	Number of participants invited (and accepted)
The Freshness of Water	Generic	N/A	N/A	43
The Clean Water Experiment	Generic	N/A	N/A	667
CS–Lime	Targeted	1500	8.9%	133
CS–Lead	Targeted	1255	8.5%	91
CS–Hardness	Targeted	2384	6.8%	163

CS: citizen science.

5. The impact of two different recruitment strategies

The impact on the sample of participants

Analysis of the registration data of the five CS projects indicates a clear relationship between the recruitment strategy and the sample composition.

Diversity in age

As depicted in Table 2, in both projects with a generic invitation strategy, that is, the Freshness of Water and the Clean Water Experiment, relatively few seniors registered, and the share of the people who registered over age 65 was, respectively, 4% and 6%. At the same time, we find that the same projects resulted in, respectively, 15% and 10% registrations from citizens aged 24 or younger. Also, the chi-square analysis showed that the age distribution of the registered people was significantly dependent on the invitation strategy (p < 0.001). A plausible explanation for the relatively many young volunteers is that both projects recruited volunteers by means of social media. Instead, the three CS projects using the targeted invitation strategy, in which specific households were asked to participate, recruited no volunteers of 24 years old or younger, and relatively many seniors; the percentage of the people who registered over age 65 ranges between 16% and 26%.

OPEN IN VIEWER

Table 2.

Diversity in age, gender and education compared between five CS projects.

		Freshness of Water	Clean Water Experiment	CS–Lime	CS–Lead	CS–Hardness	Total
Age	24 or younger	15%	10%	0%	0%	0%	7%
	25–34	32%	16%	7%	17%	6%	15%
	35–44	19%	30%	13%	29%	12%	24%
	45–54	19%	26%	28%	11%	26%	25%
	55–64	12%	12%	25%	21%	32%	17%
	65 or older	4%	6%	26%	16%	24%	11%
Sex	Male	34%	54%	72%	62%	67%	57%
	Female	66%	41%	28%	38%	33%	40%
Education	Primary school	0%	6%	2%	1%	1%	4%
	Lower secondary professional education	5%	1%	10%	3%	10%	4%
	Intermediate vocational education	7%	9%	29%	9%	33%	15%
	Pre-university education	6%	9%	13%	10%	10%	9%
	Bachelor’s degree	35%	29%	35%	33%	33%	31%
	Graduate degree	47%	46%	11%	37%	15%	37%

CS: citizen science.

The total sample is 1054.

Diversity in education

Similar to most CS projects (Brossard et al., 2005; Overdevest et al., 2004; Raddick et al., 2013), the two CS projects in this study that used a generic recruitment strategy are characterized by a relatively high percentage of citizens with high education. Respectively, 47% and 46% of the volunteers even had a graduate degree as their highest level of education; see Table 2. ⁴ The share of people with a lower education is proportionately low, and for, respectively, 7% and 9% of the respondents, intermediate vocational education was the highest completed education. The projects that used the targeted recruitment strategy show that the overrepresentation of highly educated citizens in research projects can be overcome. In fact, we find that the distribution of education levels of the participants was significantly related to the type of invitation strategy of the project (p < 0.001 with chi-square test). Both the CS and Lime and the CS and Hardness projects managed to recruit relatively many volunteers with an intermediate vocational education as their highest level of education, 29% and 33%, respectively. In the same projects, the percentage of respondents holding a graduate degree is merely 11% and 15%. As depicted in Figure 1, this analysis shows that a targeted invitation strategy other than a generic invitation to the general public has the potential to result in a higher sample diversity.

OPEN IN VIEWER

Figure 1.

Diversity in education compared between the two different invitation strategies. The relative share of each educational level is depicted on the y axis, the relative share of each invitation strategy is depicted on the x axis.

Next to diversity in age and education, also achieving gender diversity has long been a guiding principle in EC policy (European-Commission, 2018). Table 2 depicts the diversity in gender, demonstrating that the majority of respondents is male (57% versus 40% female, and 3% unknown). The reason why projects with the targeted recruitment strategy attracted relatively more male participants may relate to the fact that the invitations were sent to the drinking water companies contact persons, which are the persons in charge of paying the bill. It is not unlikely that these are relatively often men. However, as we have no reason to suppose that male volunteers respond differently than female ones to the participation in CS projects, this study does not consider gender as a variable in establishing meaningful connections. An analysis of ethnicity would also be interesting, but could not be included in this study because the respective data are unavailable. In order to avoid putting off potential participants, more sensitive questions, such as those regarding race/ethnicity and income level, were not included in the registration survey.

Motivation

The motivation of participants to engage in the five water CS projects shows a clear pattern across age groups (p < 0.001 with chi-square test; Figure 2). All motivation categories, except acquiring new knowledge, were related to age class (p < 0.05 with chi-square test on each motivation category). The interest in the particular topic of the project is more important for the older participants. Younger participants are more strongly motivated by the fun element or the opportunity to carry out measurements by themselves, and less often motivated by the special interest in the subject (p < 0.01 with chi-square test on this motivation).

OPEN IN VIEWER

Figure 2.

The motivation of participants compared across age groups. The relative share of each motivation is depicted on the y axis, the relative share of each age class is depicted on the x axis. The total sample is 1054. A box is marked with a plus (+) when standardized residuals are greater than 2, and marked with a minus (-) when standardized residuals are smaller than 2.

The motivation of participants also showed significant patterns across education levels (p < 0.01 with chi-square test). People with the lowest and highest education levels tend to be motivated by the fun element (p < 0.001 with chi-square test on this motivation). One remarkable observation is that participants with a low level of education (primary school only) perceive CS more often than others as a learning opportunity.

6. The value of diversity

Effects of participation

In this section, we will investigate the effects of participation on confidence in institutions, confidence in water quality, water awareness, CS motivation and learning. In the evaluation surveys of the various projects, a vast majority of all respondents indicated that their confidence in the (drinking) water company that organized the CS project had increased (52%) or remained stable (43%). In this matter, we did not find a clear difference across age groups or education levels (p > 0.05 with chi-square test). The few participants (2%) whose confidence decreased all relate to occasions where lead was found in the drinking water system in the CS and lead project. The confidence in the quality of (drinking) water had increased for about half of the participants (51%), and for another 44% their level of confidence in the quality of (drinking) water remained equal. The increase in the confidence level was significantly correlated with age class (rho = 0.13, p < 0.05 with rank correlation test). We found marginal age dependency on confidence in water quality (p = 0.061 with Fisher’s exact test). Young people (<35) tended to have decreased confidence, whereas the strongest increase in confidence was found in the oldest participant group. There was no clear relationship between the confidence in water and educational background (p = 0.512 with chi-square test). When asked about a change in water awareness, a majority of participants (70%) indicated that this had increased; see Table 3. We did not find a clear pattern in relation to educational background (p = 0.563 with Fisher’s exact test) or age class (p = 0.255 with chi-square test).

OPEN IN VIEWER

Table 3.

The effect of participation on water awareness compared across education levels and age groups.

		Fully agree	Agree	Neutral	Disagree	Fully disagree	Do not know
Education	Primary school	33%	17%	50%	0%	0%	0%
	Lower secondary professional education	30%	30%	30%	0%	0%	10%
	Intermediate vocational education	14%	50%	24%	12%	0%	0%
	Pre-university education	14%	52%	29%	5%	0%	0%
	Bachelor’s degree	19%	49%	21%	9%	3%	0%
	Graduate degree	23%	51%	18%	6%	1%	1%
Age	24 or younger	29%	57%	14%	0%	0%	0%
	25–34	26%	57%	4%	9%	0%	4%
	35–44	11%	53%	29%	2%	4%	0%
	45–54	25%	43%	25%	7%	0%	0%
	55–64	15%	52%	25%	8%	0%	0%
	65 or older	20%	43%	18%	14%	2%	2%

Depicted are the answers to the statement ‘As a result of participation, my awareness has increased’. The total sample size is 250.

Most participants (87%) responded positively to the statement ‘I have experienced participation in this project as a learning experience’. Again, the differences across age groups and educational backgrounds are very small (p = 0.257 and 0.156 with the chi-square and Fisher’s exact test, respectively). This indicates that from an educational perspective, it is meaningful to attract a diverse sample of participants. When asked whether they would consider participating again in a study in the water domain, a vast majority (90%) of participants responded positively. Although not significantly related to age class (p = 0.428 with chi-square test), we observed a pattern in which younger participants are relatively less eager to participate again (79% of the participants aged 24 or younger stated a willingness to participate in a new CS study in the future). This observation is in line with the larger share of dropouts in younger age groups. The willingness to participate again was higher in individuals with intermediate education and up; compared with the other participants, we found that participants with a lower secondary professional education less often say they would consider future participation (p = 0.034 with Fisher’s exact test).

7. Conclusions and implications

From a science policy standpoint, a powerful reason for investing in CS is the potential contribution of CS to the valorisation of knowledge, that is, to create more societal value from science. A crucial condition for CS projects to deliver on this promise is the participation and active contribution of a diverse sample of citizens. Based on a meta-analysis of five recent CS projects in the water domain, this article explores the possibilities of working with such a diverse sample, by investigating the recruitment of citizens, their active participation and the effects of participation.

We conclude that, from a valorisation perspective, it is worth the effort to attract a diverse sample of participants in CS. The meta-analysis of five CS projects presented in this article suggests that people across all age groups and education levels can participate meaningfully in CS projects, in the sense that they are moved or affected by their participation. The current literature is ambivalent about the possibilities of citizens acquiring knowledge about the research topic or about science in general (Bonney et al., 2016; Jollymore et al., 2017). In this article, we did not analyse scientific literacy as such, but we did find effects of participation on confidence and awareness in participants of all ages and education levels. The majority of participants expressed increased confidence in the water company and in water quality, as well as greater water awareness. This indicates that their CS participation has changed the relationship with the object of study. Apart from a relationship between age and the change in confidence in the quality of water, we did not find significant variations between education levels or age groups. Our finding that lower educated people can also participate meaningfully in CS resonates with Den Broeder et al.’s (2017) findings that citizen scientists with lower education levels can benefit from their participation in terms of self-confidence, social skills and cross-cultural network.

The second conclusion of this article is that it is possible to recruit a diverse sample of citizen scientists, and that a homogeneous group, consisting primarily of highly educated CS volunteers, should no longer be seen as inevitable. The three projects we have analysed, which used a targeted recruitment strategy, managed to attract a significantly higher diversity in participation. Citizens of older age and citizens with lower level education were much better represented than in the projects with a generic recruitment strategy. This shows that sending out personal invitations to a random sample of the population can help to attain a diverse sample of participants, as opposed to the scattergun approach of advertising on mass media or social media (West and Pateman, 2016). This is notable because CS projects are known to attract mainly highly educated participants (Brossard et al., 2005; Crall et al., 2013; Overdevest et al., 2004). It cannot be excluded that the specific focus of the projects and the contribution required from the citizens have influenced the composition of the sample of participants, which could be considered as a limitation in this study. We would argue, however, that this influence is limited, as all five CS projects were conducted in the Netherlands, around the same time, and all dealt with (drinking) water topics.

Although the recruitment of a diverse sample of volunteers indeed seems very possible, the active participation of all participants remains a concern. In the younger participants, we found a higher dropout during the projects, and also a lower willingness for a repeat participation. This suggests that additional efforts are required during the project to prevent the loss of interest (Conrad and Hilchey, 2011) and to keep these more volatile groups motivated.

Not all CS projects are aimed at valorisation. Particularly, instrumental CS projects of the ‘contributory’ type may well be oriented at generating large datasets or increasing fundamental understanding, without generating social impacts. However, it seems more than reasonable to assume that a large share of all CS efforts will be oriented at valorisation. For these projects, we derive some practical recommendations from our analysis. First, we would encourage project designers to consider a targeted recruitment strategy rather than an open call if they intend to attract a diverse sample of participants. Second, in order to sustain the activity of younger participants, project designers may explore possibilities to ask more commitment at the start and to invest more in face-to-face contact with their (younger) participants. Third, given our finding that younger participants tend to be motivated less by the specific topic of study and more by the possibility of contributing to science in general or by the fun aspect of CS, project designers interested in valorisation may consider strengthening these aspects (e.g. by site visits or social media activities), or stressing them more in the communication with participants.

Furthermore, we would argue that the discourse about CS deserves to be refined, as many contributions to the CS literature are insufficiently precise about the rationales for CS. We recommend to policy-makers and subsidy-granting agencies working with CS to make the valorisation goals of CS explicit, including the underlying goals and assumptions. In this manner, scholars and institutions designing CS projects can purposefully choose whether to take measures to enhance diversity of participation, and to make a contribution to the valorisation of knowledge.

As a final remark, it should be noted that in addition to diversity, certainly also a critical mass is necessary to fully extend the social impacts of science. Although the number of CS participants is not explicitly covered in this article, it is interesting to note that from the targeted invitation response rate data depicted in Table 1, it can be inferred that the potential of CS volunteers nationwide may be in the millions. In sum, we believe that if some of the necessary steps outlined in this article are taken, the valorisation potential by involving citizens certainly shows substantial promise.

Supplemental Material