Crowdsourcing a mixed systematic review on a complex topic and a heterogeneous population: Lessons learned

1. Introduction

Systematic reviews (SR) are commonly used to inform evidence-based decision- and policy-making in multiple areas. However, the selection of relevant studies is burdensome, given the continuously growing number of publications (Brown & Allison, 2014; Krivosheev et al., 2017; Mortensen et al., 2017). Interestingly in the 19th century and the 20th century, the invention of abstracts and the implementation of bibliographic databases were aimed to deal with this very same issue, respectively. Crowdsourcing may be their 21st century counterpart. Specifically in SR on new research topics with no established classification of knowledge, this is even more challenging (Weiss, 2016). Librarians and information professionals are often involved in SR as expert searchers, knowledge managers and analysts (Cooper & Crum, 2013). Therefore, it is important for information professionals supporting or participating in SR to know about new SR methods such as crowdsourcing.

Crowdsourcing “draws on a large pool of people who individually make small contributions that add up to big efforts” (Strang & Simmons, 2018, p. 1). This innovative method of data acquisition is extensively utilized in many research fields (Samimi & Ravana, 2014). Specifically, crowdsourcing has been used as an “economical and time saving means to evaluate large bodies of published literature” (Brown & Allison, 2014, p. 6). In SR, crowdsourcing can overcome the challenges faced by small teams and potentially improve the overall efficiency (Ranard et al., 2014; Strang & Simmons, 2018). This commentary is aimed to summarize lessons learned from our experience using crowdsourcing to select relevant studies in the context of a SR on patients with complex care needs (PCCN) in primary health care (Bujold et al., 2017). Similar to other new research topics, PCCN-related studies are “broad, fragmented and growing quickly” which makes literature reviews more difficult (Weiss, 2016, p. 5). The following sections are presented using the Conceptual Foundations of Crowdsourcing proposed by Pedersen et al. (2013): problem, process, governance, people, technology, and outcome.

2. Problem

SR typically synthesize results of included studies on well-defined issues, homogeneous populations and simple programs. In contrast, conducting a SR on PCCN presents two main challenges: a new ill-defined topic and a heterogeneous population. PCCN often suffer from combinations of multiple chronic conditions, mental health problems, drug interactions and social vulnerability, which can lead to the overuse, underuse or misuse of healthcare services (Martello et al., 2014; Pluye et al., 2014). Moreover, the PCCN experience is exacerbated by interactional issues and difficulties in decisions-making process involving different stakeholders such as patients, caregivers and practitioners (Bujold et al., 2017). Our objective was to systematically assess decisional needs of PCCN from the literature, i.e., identify difficult decisions experienced by stakeholders.

Additional challenges encountered in this SR were related to identification and selection of potentially relevant studies. PCCN and PCCN-related difficult decision-making are idiosyncratic concepts and can be conceived differently by doctors, patients, caregivers, and researchers due to different knowledge and lived experience. The absence of established classifications of knowledge makes it difficult to identify studies presenting relevant information and interpret it (Weiss, 2016). We worked with specialized librarians to establish a comprehensive search strategy with high recall to capture research studies looking at difficult decision-making in a population with at least one PCCN characteristic (interactional issues, comorbidities, mental health issues, drug interactions, social vulnerability, or healthcare services overuse/underuse/misuse). Establishing clear and comprehensive eligibility criteria was an iterative and time-consuming process.

3. Process

Two specialized librarians conducted a comprehensive search in MEDLINE (Ovid), Embase (Ovid), PsycINFO (Ovid), CINAHL (EBSCOhost) and Social Sciences Citation Index. All results were imported into EndNote, where duplicates were removed, resulting in 8616 potentially relevant unique studies. These studies were screened by at least, two independent reviewers. To do this, all studies were imported into a specialised online software (DistillerSR) and two reviewers started the work (first and third author). However, it was too time consuming and we quickly realized that an alternative solution was needed. Thus, we turned to crowdsourcing which was applied at two levels. Our project is an example of “direct crowdsourcing” where one “requester” recruited participants, crowd members, to complete specific screening tasks (Weiss, 2016).

Motivation and engagement are key success factors in a crowdsourced project (Strang & Simmons, 2018). In our SR, weekly email updates were sent by one of the co-authors to encourage crowdreviewers. Crowdreviewers co-researchers were motivated (self-interest), and external crowdreviewers (not co-researchers on the project) were compensated for their time using gift cards (20$ per 100 Level 1 abstracts, and 1$ per one Level 2 full-text). Six crowdreviewers had previous experience using DistillerSR; those who did not, were trained.

The quality control of crowdsourced work is a common concern (Samimi & Ravana, 2014; Strang & Simmons, 2018; Weiss, 2016) and is of particular importance in SR. For each level, the software allowed the first author to verify the selection bias of crowdreviewers, specifically for two concepts due to the lack of established knowledge classification (PCCN and Decisional needs). For each crowdreviewer, sets of excluded/included studies were randomly selected and checked by the first author. Crowdreviewers who excluded potentially relevant studies or included irrelevant ones were met individually for conceptual clarification. While this process prevented some selection biases, it also contributed to the establishment of an intersubjective knowledge classification.

A strength of the crowdsourcing process is that crowdworkers “give varying meaning to a single fact or question” (Lebraty & Lobre, 2013, p. 20). As Surowiecki stated in his book on the “Wisdom of the crowd”, collective intelligence may result in better decision-making compared to individual decisions (Surowiecki, 2007). Our team was composed of 20 crowdreviewers who had different levels of (a) experience and familiarity with reviewing scientific articles, (b) using a specialised SR software, and (c) knowledge of complex care needs. They were researchers, graduate students and clinicians from different disciplines (family medicine, nursing, occupational therapy, epidemiology, and social sciences). This diversity was advantageous, leveraging different viewpoints on the complexity of care needs: medical, nursing, occupational, populational, social, and combined viewpoints (some crowdreviewers had multidisciplinary training and working experience). Moreover, some crowdreviewers personally knew a PCCN and were aware of the patient’s viewpoint.

6. Technology

Online technology is necessary to enable and facilitate crowdsourcing (Samimi & Ravana, 2014; Strang & Simmons, 2018; Thomas et al., 2017; Wazny, 2017). In our experience, the DistillerSR software was very helpful, user-friendly and available online 24/7. The technical help was accessible and effective in a timely manner. The software was essential to manage the crowdsourcing in three main aspects: inclusions, exclusions and conflicts. At Level 1, the crowdreviewers appreciated the software’s highlighting function of predefined keywords, a function also described as time-saving in other crowdsourcing projects (Strang & Simmons, 2018).

7. Outcomes

The main benefit of crowdsourcing was the diversity of crowdreviewers’ viewpoints. Crowdsourcing resulted in the selection of 156 potentially relevant studies to be included in the synthesis. The crowdsourced selection took place over a period of six weeks (June 19 to July 31, 2017). In total, crowdworkers spent 360 hours in Level 1 and Level 2 selection. Level 1 selection was done during the last two weeks of June, and required about 145 hours of crowdwork (8616 abstracts × 2 crowdreviewers × 30 seconds; the average time to include/exclude an abstract being about 30 seconds). Level 1 crowdworking time was about four hours/week/crowdreviewer on average. Level 2 took place over the month of July, and required about 215 hours of crowdwork (1293 full-texts × 2 crowdreviewers × 5 min; the average time to include/exclude a full-text being about 5 min). Level 2 crowdworking time was about 5 hours/week/crowdreviewer on average. Indeed, two experienced researchers working almost full-time could review about 8700 abstracts (e.g., 100/hour) and 1300 full-text papers over six weeks (6 weeks × 30 hours/week × 2 crowdreviewers = 360 hours); this would have been feasible but burdensome.

8. Conclusion

In line with other crowdsourced literature reviews, our experience confirms that crowdsourcing can be useful in SR with a large number of studies such as mixed studies reviews (Hong et al., 2017), and reviews of studies on ill-defined domains and involving heterogeneous populations. Similar to what is reported in the literature (Lebraty & Lobre, 2013), we experienced that crowdsourcing adds value to the creation and production of knowledge via the diversity and independence of the crowd’s members. Even though some items proposed by Cox et al. (2015) to assess the outcomes of crowdsourcing are not appropriate for assessing our work (e.g., the dimension “data value”), we addressed most of them in this commentary (see Table 2). Furthermore, this framework can be used by information professionals to assess and improve the crowdsourcing activities they are involved in.

In our SR, crowdsourcing faced challenges and led to benefits. For example, while building training material and training crowdreviewers took time, the diversity of their viewpoints, comments and questions helped to build a clear manual for selecting abstracts and full-texts on the PCCN complex issue. By way of another illustration, crowdreviewers had different levels of expertise, which was challenging, while the diversity of crowdreviewers contributed to establish an intersubjective knowledge classification of idiosyncratic concepts related to PCCN and their decisional needs. Given the heterogeneity of the population of PCCN and the newness of related concepts, the interweaving of the crowdreviewers’ viewpoints allowed us to develop comprehensive and clear definitions.