Facilitators and barriers associated with the implementation of an innovative cross-disciplinary monitoring of the scientific literature: The Collaborative eBibliography on Mixed Methods (CeBoMM)

Abstract

Mixed methods (MM) involve combining qualitative (QUAL) and quantitative (QUAN) methods in program evaluation, primary research, and literature reviews. MM are being increasingly used in health and social sciences (in multiple fields and in an inter-field manner). Over the years, several strategies to integrate QUAL and QUAN phases, results, and data have been proposed. For MM teachers, one of the challenges is to explain specific MM strategies and their combinations, find current illustrative examples for trainees, and identify emerging innovative MM strategies. Our project is aimed at identifying and measuring the importance of facilitators and barriers associated with the implementation of an innovative cross-disciplinary monitoring of the research literature: the Collaborative eBibliography on Mixed Methods (CeBoMM). Results will facilitate CeBoMM implementation, and can contribute to MM teaching and learning, thereby, helping MM teachers and their trainees worldwide. Ultimately, CeBoMM can be adapted to be used by teachers in other academic areas and those interested in collaborative information monitoring.

Keywords

Mixed methods collaborative monitoring of research trends eBibliography

1. Introduction

Mixed methods (MM) involve combining qualitative (QUAL) and quantitative (QUAN) methods in program evaluation, primary research, and literature reviews (Creswell et al., 2017; Greene, 2008; Hong et al., 2017; Johnson et al., 2007; Pluye & Hong, 2014; Tashakkori & Teddlie, 2010). MM are being increasingly used in health and social sciences (in multiple fields and in an inter-field manner). Over the years, several strategies to integrate QUAL and QUAN phases, results, and data have been proposed. For MM teachers, the main challenge is to explain specific MM strategies and their combinations, find current illustrative examples for trainees (publications with detailed methodological description), and identify emerging MM strategies.

As a contribution for addressing this challenge, the purpose of our research project is to explore facilitators and barriers associated with the implementation of an innovative cross-disciplinary monitoring of the MM literature in health and social sciences: the Collaborative eBibliography on Mixed Methods (CeBoMM). Usually, a bibliography is a collection of works chosen by a compiler. In contrast, CeBoMM will contain an overview of MM and an annually annotated collection of selected research abstracts reporting MM empirical studies (with bibliographic records and links to full-text documents). The selection of publications will be crowdsourced to a ‘crowd’ of MM teachers. Selected studies will illustrate all types of (a) combinations of worldviews used in MM, (b) MM research designs, (c) combinations of strategies for integrating QUAL and QUAN phases, results and data in primary research, and (d) syntheses for integrating QUAL and QUAN evidence in mixed studies reviews.

In the next sections, we will present the definition and key aspects of MM that justify the proposed project (pluralism of worldviews, MM research designs, and types of MM integration). We will then define our innovative collaborative monitoring of the literature, and describe the methods used for conducting this project, expected results and knowledge translation plan. While this protocol is centered on MM, information professionals and teachers may consider this type of innovative collaborative monitoring of trends as it can be transferable to other areas and across disciplines.

2. Rationale

In this project, to be considered MM, studies have to meet the following criteria: (a) at least one QUAL method and one QUAN method are combined; (b) each method is used rigorously in accordance to the generally accepted criteria in the area (or tradition) of research invoked (e.g., ethnography and randomized controlled trial); and (c) the combination of the methods is carried out at minimum through a MM design (defined a priori, or emerging) and the integration of the QUAL and QUAN phases, results, and data (Creswell et al., 2017; Pluye et al., 2018). The QUAL and QUAN methods can also (but not necessarily) be combined with regard to the data collection (mixed instrumentation), the literature review (mixed studies review justifying the MM research questions and design), and the MM team members’ interpretations of sciences in terms of epistemology, ontology, teleology, and methodology (hereafter termed worldview).

Conversely, the following types of research are not considered MM in this project: (a) a QUAN method with a collection and analysis of qualitative information that does not consist of research data because it does not refer to a QUAL research methodology and method, (b) a QUAL method with a collection and analysis of quantitative information that does not consist of research data because it does not refer to a QUAN research methodology and method, (c) a combination of QUAN methods, (d) a combination of QUAL methods, and (e) the juxtaposition of QUAL and QUAN methods (similar to two separate studies) without integration of QUAL and QUAN approaches, questions, designs, instrumentations, phases, results, and data.

2.1 Pluralism of worldviews in MM

In 2003, Teddlie and Tashakkori affirmed the co-existence of different worldviews in MM (Teddlie & Tashakkori, 2003). This pluralism is illustrated by numerous publications (Niglas, 2010). MM research team members may share a common worldview, and explicitly or tacitly agree with respect to the epistemological, ontological, teleological, and methodological foundations of their work (Pluye & El Sherif, 2017; Ridde & Dagenais, 2012). When a team includes MM researchers whose worldviews differ, the combination of methods requires epistemological, ontological, teleological, and methodological discussions. For example, the team can seamlessly combine five common worldviews that recognize QUAL and QUAN methods such as Campbell’s postpositivism, Hacking’s social constructionism, pragmatism, critical realism, and critical theories (Campbell, 1988; De Waal, 2005; Hacking, 1999; Sayer, 2000; Tyson, 2014). Finding examples of pluralism is difficult for individual MM teachers as worldviews are rarely discussed in publications reporting empirical studies. The proposed project and CeBoMM may help solve this issue, and select publications reporting MM empirical studies with a detailed discussion of worldviews.

2.2 Common types of MM designs

A common classification of MM is based on two types of designs (sequential and convergent designs) and three main variants (multiphase, multilevel, and multiphase-multilevel) (Creswell & Plano Clark, 2017). First, sequential designs use a QUAL method followed by a QUAN method (e.g., QUAN methods and results are used to statistically generalize some QUAL results), or a QUAN method followed by a QUAL method (e.g., QUAL methods and results are used to interpret some QUAN results). In any sequential design, Phase-1 results inform Phase-2. Assuming that a research project can be conceptualized as an organizational process (e.g., a collective project involving QUAL and QUAN researchers), the literature on organizations (project management) provides a useful definition of such sequence (hereby defining the concept of integration in MM sequential designs). Inspired by Van de Ven (Van de Ven, 1992), a sequence consists of a developmental change in the project’s orientation over time (results of a first data collection and analysis [phase-1] inform a second data collection and analysis [phase-2]), and a cognitive transition of the researchers at the time of change (from QUAL to QUAN, or from QUAN to QUAL).

Second, convergent designs combine the QUAL and QUAN methods during data collection and analysis, while the QUAL and QUAN methods are often (but not necessarily) concomitant. The literature on organizations and processes of collective decision-making (e.g., decisions made by a team of researchers using MM) provides a useful definition of convergence (thereby defining the concept of integration in MM convergent designs). Inspired by Langley et al. (Langley, Mintzberg, Pitcher, Posada, & Saint-Macary, 1995) convergence is defined as a process of progressive, successive, and constant improvements during the collection and analysis of QUAL and QUAN data (convergence of data), or the interpretations of results (convergence of results). The researchers work in a prospective, non-linear way, guided by a cognitive representation of the additional data, or databases, or analyses of data, or results to be created.

Third, the variants of these designs simply involve multiplying the phases or levels of data collection and analysis. The multiphase design includes three sequential phases (e.g., QUAL then QUAN then QUAL) or more (Lisle, 2013). The multilevel design includes two levels of analysis (e.g., QUAN at the individual level and QUAL at the organizational level) or more (Dagenais et al., 2008). This design is based on the convergence of the results of the analyses carried out at each level. Furthermore, the two variants can be combined in a multiphase-multilevel design (Youngs & Piggot-Irvine, 2011).

In MM designs (sequential, convergent, multiphase, multilevel, or multiphase-multilevel), the QUAL designs most commonly combined with a QUAN design are descriptive or interpretative qualitative research, exploratory case studies, ethnography, grounded theory, phenomenology, and life stories or biographies (Schwandt, 2007). The QUAN designs most commonly combined with a QUAL design are descriptive surveys (e.g., prevalence or incidence studies), non-randomized studies (e.g., analytical survey, or cohort, or case-comparison, or quasi-experiment), and randomized controlled trials (Porta, 2008). Special mention can be made for case study and grounded theory that are usually QUAL, but may be QUAN or MM (Johnson et al., 2010; Yin, 2006). For each research area, the proposed project and CeBoMM will lead to select current examples of all types of MM designs with detailed method description. This will help MM teachers as finding such examples every year for updating MM courses is currently a daunting task for individual teachers (seeking the MM literature by themselves, alone or with a teaching assistant).

2.3 Three types of integration in MM

In 2010, a 13-dimension general review of MM analyses was published (Onwuegbuzie & Combs, 2010). Apart from this review, methodological articles and books propose single strategies (each strategy being presented as a necessary and sufficient process to obtain results); publications are usually prescriptive (not tested empirically) and limited to few combinations (e.g., one-design and one-strategy combination). We have analyzed the most cited methodological publications, and grouped strategies into three main categories of specific strategies: those that (a) connect the QUAL and QUAN phases, (b) compare the results of QUAL and QUAN, and (c) assimilate the QUAL and QUAN data (Pluye et al., 2018).

These categories refer to the three common types of integration of QUAL and QUAN methods; they integrate previous terminology (Bazeley, 2009; Creswell et al., 2017; Greene, 2007; Guetterman et al., 2015; Teddlie & Tashakkori, 2003, 2009, 2010) and have been defined using harmonization principles (International Standards Organization (ISO), 2009; Roche, 2012). They are not mutually exclusive (i.e., they can be combined) and are not hierarchically ordered; for example, phase connection does not refer to a higher (or lower) degree of integration compared to results comparison or assimilation of data.

2.3.1 Connection of phases

The complementarity principle is derived from the literature suggesting that the worldviews associated with the QUAL methods are different and separate from those associated with the QUAN methods. Thus, methods for collecting and analyzing QUAL and QUAN data must be kept separated. QUAL and QUAN methods and results are presented separately in the MM publications, and the complementarity of the QUAL and QUAN results is described. Integration (cognitive transition) occurs during the connection between two phases (e.g., between a QUAL and a QUAN phase).

2.3.2 Comparison of results

The principle of dialectical tension comes from the literature suggesting that the worldviews associated with QUAL and QUAN methods are different and interdependent (their juxtaposition generating creative tensions leading to discovery and innovation). Thus, the QUAL and QUAN data collection and analysis methods are separated, or interconnected, and the results are interconnected using a comparison process. The similarities, differences and contradictions between QUAL and QUAN results are explained (guided by a cognitive representation of the results to be created). For instance, discrepancies between the QUAN and QUAL results are mentioned and discussed in the MM publications.

2.3.3 Assimilation of data

The third principle, unification, focuses on a worldview (or an approach such as participatory research) associated with the QUAL and QUAN methods. It corresponds to two streams of thought: on the one hand, MM can address research questions and mobilize theories that unify the use of QUAL and QUAN methods (independently of the worldviews); on the other hand, several worldviews directly allow the integration of QUAL and QUAN methods in MM (unification on a shared worldview). This principle justifies the assimilation of the data (guided by a cognitive representation of the results to be created). The QUAL and QUAN data can be transformed into a single QUAL (themes) or QUAN (variables) form, or merged on a case-by-case basis.

As mentioned for worldviews and designs, CeBoMM will help individual MM teachers access a list of current examples of MM integration strategies, including emerging new strategies. For each strategy type, examples will be selected by the crowd of MM teachers (l’union fait la force).

2.4 The value of collaborative monitoring of scientific literature

Awareness of current research evidence is fundamental to evidence-based practice and evidence-informed decision/policy-making. However, keeping up to date with scientific literature remains a major challenge due to information overload and time constraints (Pontis et al., 2017). Services and tools exist to continuously monitor scientific literature (e.g., alerting services) (Badran et al., 2015), but the growing number of options further contributes to information overload (Barr, 2006). This is particularly true for those working in complex multidisciplinary fields such as MM in health and social sciences. Not only is the body of knowledge ever-growing and rapidly-changing, MM teachers need to cast their nets wide to identify few relevant studies.

Researchers and practitioners struggle with keeping up to date (Badran et al., 2015; Barr, 2006; Grad et al., 2014; Pluye et al., 2013; Pluye et al., 2013; Pluye et al., 2014; Pontis et al., 2017). In health sciences for instance, the growing involvement of a diversity of stakeholders (decision/policy-makers, patients, caregivers, relatives and practitioners) in MM research makes the need for a new solution even more critical. Using collaboration to keep up to date may contribute to address this issue. By working together and sharing monitoring and filtering tasks, the individual burden may be reduced, making it easier and faster to stay current. In line with the Canadian Strategy for Patient Oriented Research (SPOR), collaborative filtering may aid MM stakeholders with less research experience such as patients, caregivers and relatives who work in partnership with researchers at all research stages.

Figure 1.

eSRAP login interface.

Figure 2.

eSRAP reading and rating interface.

The proposed project will be based on eSRAP ${}^{\copyright}$ (see Figs 1 and 2), a collaborative research trend monitoring system developed by the Method Development component of the Quebec SPOR SUPPORT Unit (Tang et al., 2015). It monitors the latest scholarly publications as they become indexed in bibliographic databases and uses crowdsourcing to identify peer-appraised research trends. Crowdsourcing has been widely used in numerous areas and disciplines (Lebraty & Lobre, 2013; Pedersen et al., 2013; Surowiecki, 2007), also in health sciences (Créquit et al., 2018; Ranard et al., 2014; Wazny, 2017). For example, it has been used in literature reviews on various health topics (Brown & Allison, 2014; Bujold et al., 2018; Mortensen et al., 2017; Thomas et al., 2017; Weiss, 2016). However, according to literature reviews crowdsourcing has not been used for monitoring research trends yet, suggesting that CeBoMM could be seen as an innovation (Greenhalgh et al., 2004).

A draft CeBoMM has been developed with and for graduate students of a graduate course offered since 2007 in the Department of Family Medicine at McGill University, Montréal, Canada (FMED 672 ‘Applied MM in Health Sciences’). The present project will explore factors associated with the implementation of CeBoMM in the larger community of MM teachers worldwide, working in cross-disciplinary context in all health and social sciences. Results may be transferable for implementing such collaborative research trend monitoring system in other scientific communities as an eSRAP-diy (do it yourself) version will be available in 2020.

Therefore, our general goal is to design CeBoMM, i.e., a dynamic collaborative online anthology of mixed methods research and systematic mixed studies reviews in-patient oriented research (health and social sciences). Our specific objectives are twofold: (1) to identify barriers and facilitators for engaging in collaborative monitoring and use of CeBoMM (such as motivations and expected benefits) from the perspective of MM teachers, and (2) measure the importance of these factors among MM teachers.

3. Methodology and methods

A two-phase sequential exploratory MM design will be used. Combining the methodologies will allow us to document MM teachers’ perceptions and experiences with monitoring and filtering the MM literature. Ethical approval will be obtained from the Institutional Review Board of the Faculty of Medicine at McGill University. In preparation for this project, a census of MM teachers will be conducted by specialized librarians using a Web search, e.g., through the MMIRA (Mixed Methods International Research Association) and affiliated organizations’ websites such as MMF (Méthodes mixtes francophonie).

3.1 Phase 1

A qualitative multiple case study with a purposeful maximum variation sample of participants (based on demographics of MM teachers worldwide) will be conducted (Yin, 2014). We will start with 10 participants and will add sets of 3 participants until we reach data saturation (between 15 and 30 participants are usually sufficient). Each participant will constitute a case. Data collection will include correspondence with participants, semi-structured interviews and participants’ MM training material, e.g., syllabi. Interviews will include only few open questions and probes to help participants express their own experience and views about monitoring the literature for teaching MM. Then, semi-structured questions will inquire about participants’ views about collaborative monitoring (CeBoMM). These questions will be informed by an implementation framework (Greenhalgh et al., 2004). All interviews will be digitally recorded and transcribed. All textual data will be analyzed using inductive and deductive thematic analysis to identify participants’ patterns for monitoring and filtering issues, and views on CeBoMM implementation factors (Fereday & Muir-Cochrane, 2006). MM integration (connection): Phase-1 QUAL results will inform Phase-2 QUAN data collection.

3.2 Phase 2

A quantitative cross-sectional survey of the population-wide MM teachers will be conducted. Using Dillman’s survey guidelines, structured questions will be derived from phase-1 results (Dillman et al., 2009). An invitation to participate will be emailed to all identified MM teachers with a link to an online consent form and questionnaire. All quantitative data will be analyzed using descriptive statistics, measuring the frequency of MM teachers’ monitoring and filtering patterns, and the relative importance of CeBoMM implementation factors. Inferential statistics will be used for estimating associations between patterns and factors when appropriate. Usual statistical techniques will be applied to control for the non-response source of bias (Lavrakas, 2013). Integration (comparison): QUAL and QUAN results will be compared and interpreted in terms of corroboration and divergence (Pluye et al., 2009).

4. Expected results and knowledge translation

Results will provide needed understanding of the barriers and facilitators, as well as potential benefits associated with collaborative monitoring of MM. They will be used for implementing CeBoMM. The contribution to knowledge represented by these results will be also valuable to information scientists and practitioners, specifically those studying and providing monitoring services (information specialists, knowledge providers and system developers). In addition, results will be relevant to other researchers’ and teachers’ groups, who require and struggle with monitoring trends in their fields; e.g., they can be transferable to other communities of teachers (teaching other methodologies and methods, and cross-disciplinary topics). Ultimately, these results can inform how we prepare future scholars to overcome the challenges of monitoring research trends.

Footnotes

Acknowledgments

The initial version of eSRAP has been sponsored by the Quebec SPOR SUPPORT Unit, the National Research Council of Canada (NRC), and other professional and philanthropic organizations. The second author holds a Doctoral Fellowship Award from the FRQ-SC (Société et Culture).

References

Badran,

. Pluye,

, & Grad,

R. M.

(2015). Advantages and Disadvantages of Educational Email Alerts for Family Physicians: Viewpoint. Journal of Medical Internet Research,17(2), e49.

Barr,

(2006). Staying alert: The wild new world of current awareness services. College and Research Libraries News,67(1), 14-17.

Bazeley,

(2009). Analyzing mixed methods data. In Andrew

& Halcomb

(Eds.), Mixed methods research for nursing and the health sciences (pp. 84-118). Chichester: Wiley.

Brown,

A. W.

, & Allison,

D. B.

(2014). Using crowdsourcing to evaluate published scientific literature: Methods and example. PLoS ONE,9(7), e100647.

Bujold,

Granikov,

El Sherif,

, & Pluye,

(2018). Crowdsourcing a mixed systematic review on a complex topic and a heterogeneous population: Lessons learned. Education for Information (Special Issue),34(4), 293-300.

Campbell,

D. T.

(1988). Qualitative knowing in action research. In Overman

(Ed.), Methodology and epistemology for social science: selected papers of Donald T. Campbell (pp. 360-376). Chicago: The University of Chicago Press.

Créquit,

Mansouri,

Benchoufi,

Vivot,

, & Ravaud,

(2018). Mapping of Crowdsourcing in Health: Systematic Review. Journal of Medical Internet Research,20(5), e187.

Creswell,

J. W.

, & Plano Clark,

(2017). Designing and conducting mixed methods research [3rd edition]. Thousand Oaks: Sage.

Dagenais,

Nault-Brière,

Dupont,

, & Dutil,

(2008). Implementation and effects of a service coordination program for youths and their families in complex situations: A mixed evaluation design. Children and Youth Services Review,30(8), 903-913.

10.

De Waal,

(2005). On pragmatism. Belmont: Wadsworth.

11.

Dillman,

D. A.

Smyth,

J. D.

, & Christian,

L. M.

(2009). Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. Hoboken: Wiley.

12.

Fereday,

, & Muir-Cochrane,

(2006). Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development. International Journal of Qualitative Methods,5(1), 1-11.

13.

Grad,

Pluye,

Repchinsky,

Jovaisas,

Marlow,

Marques Ricarte,

I. L.

, et al. (2014). Physician assessments of the value of therapeutic information delivered via e-mail. Canadian Family Physician,60(5), e258-e262.

14.

Greene,

J. C.

(2007). Mixed methods in social inquiry. San Francisco: Jossey Bass.

15.

Greene,

J. C.

(2008). Is mixed methods social inquiry a distinctive methodology?Journal of Mixed Methods Research,2(1), 7-22.

16.

Greenhalgh,

Robert,

Macfarlane,

Bate,

, & Kyriakidou,

(2004). Diffusion of innovation in service organisations: systematic review and recommendations. Milbank Quarterly,82, 581-629.

17.

Guetterman,

T. C.

Fetters,

M. D.

, & Creswell,

J. W.

(2015). Integrating quantitative and qualitative results in health science mixed methods research through joint displays. Annals of Family Medicine,13(6), 554-561.

18.

Hacking,

(1999). The social construction of what? Cambridge: Harvard University Press.

19.

Hong,

Q. N.

Pluye,

Bujold,

, & Wassef,

(2017). Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Systematic Reviews,6, e61.

20.

International Standards Organization (ISO). (2009). Terminology work – Principles and methods. Retri-eved from http://www.iso.org/iso/home/store/catalogue_tc/catalogue_detail.htm?csnumber=38109.

21.

Johnson,

McGowan,

, & Turner,

(2010). Grounded theory in practice: Is it inherently a mixed method. Research in the Schools,17(2), 65-78.

22.

Johnson,

Onewuegbuzie,

, & Turner,

(2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research,1(2), 112-133.

23.

Langley,

Mintzberg,

Pitcher,

Posada,

, & Saint-Macary,

(1995). Opening up decision making: the view from the black stool. Organization Science, 6(3), 260-279.

24.

Lavrakas,

P. J.

(2013). Presidential address: Applying a total error perspective for improving research quality in the social, behavioral, and marketing sciences. Public Opinion Quarterly, 77(3), 831-850.

25.

Lebraty,

J.-F.

, & Lobre,

(2013). Crowdsourcing: One step beyond. London: John Wiley & Sons.

26.

Lisle,

J. D.

(2013). Exploring the value of integrated findings in a multiphase mixed methods evaluation of the continuous assessment program in the Republic of Trinidad and Tobago. International Journal of Multiple Research Approaches,7(1), 27-49.

27.

Mortensen,

M. L.

Adam,

G. P.

Trikalinos,

T. A.

Kraska,

, & Wallace,

B. C.

(2017). An exploration of crowdsourcing citation screening for systematic reviews. Research Synthesis Methods,8(3), 366-386.

28.

Niglas,

(2010). The multidimensional model of research methodology: an integrated set of continua. In Tashakkori

& Teddlie

(Eds.), Handbook of mixed methods in social and behavioral research (2nd edition) (pp. 215-236). Thousand Oaks: Sage.

29.

Onwuegbuzie,

A. J.

, & Combs,

J. P.

(2010). Emergent data analysis techniques in mixed methods research: a synthesis. In Tashakkori

& Teddlie

(Eds.), Handbook of mixed methods in social and behavioral research (pp. 397-430). Thousand Oaks: Sage.

30.

Pedersen,

Kocsis,

Tripathi,

Tarrell,

Weerakoon,

Tahmasbi,

, et al. (2013). Conceptual foundations of crowdsourcing: A review of IS research. 46th Hawaii International Conference on System Sciences, 579-588.

31.

Pluye,

, & El Sherif,

(2017). La contribution des méthodes mixtes à l’intégration des interprétations des sciences. Paper presented at the ACFAS Colloquium 630 on Innovations in Mixed Methods Health and Social Sciences, Montreal.

32.

Pluye,

Grad,

Johnson-Lafleur,

Granikov,

Shulha,

Marlow,

, & Ricarte,

(2013). The Number Needed to Benefit from Information (NNBI): proposal from a mixed methods research study with practicing family physicians. Annals of Family Medicine,11(6), 559-567.

33.

Pluye,

Grad,

Levine,

, & Nicolau,

(2009). Understanding divergence of quantitative and qualitative data and results in mixed methods studies. International Journal of Multiple Research Approaches,3(1), 58-72.

34.

Pluye,

Grad,

Repchinsky,

Jovaisas,

Johnson-Lafleur,

Carrier,

, et al. (2013). Four levels of outcomes of information-seeking: a mixed methods study in primary health care. Journal of the American Society for Information Science and Technology,64(1), 108-125.

35.

Pluye,

Grad,

R. M.

Repchinsky,

Jovaisas,

Lewis,

Tang,

D. L.

, et al. (2014). ‘Better than best’ evidence? The Information Assessment Method can help information providers to use family physicians’ feedback for 2-way knowledge translation. Canadian Family Physician,60(5), 415-417.

36.

Pluye,

, & Hong,

(2014). Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annual Review of Public Health,35, 29-45.

37.

Pluye,

Kaur,

Granikov,

Garcia Bengoechea,

, & Tang,

(2018). A World of Possibilities in Mixed Methods: Review of the Combinations of Strategies Used to Integrate Qualitative and Quantitative Phases, Results and Data [invited paper]. International Journal of Multiple Research Approaches,10(1), 41-56.

38.

Pontis,

Blandford,

Greifeneder,

Attalla,

, & Neal,

(2017). Keeping up to date: An academic researcher’s information journey. Journal of the Association for Information Science and Technology,68(1), 22-35.

39.

Porta,

(2008). A dictionary of epidemiology. New York: Oxford University Press.

40.

Ranard,

B. L.

Ha,

Y. P.

Meisel,

Z. F.

Asch,

D. A.

Hill,

S. S.

Becker,

L. B.

, et al. (2014). Crowdsourcing – harnessing the masses to advance health and medicine, a systematic review. Journal of General Internal Medicine,29(1), 187-203.

41.

Ridde,

, & Dagenais,

(2012). Introduction générale à l’évaluation de programme. In Ridde

& Dagenais

(Eds.), Approches et pratiques en évaluation de programme (pp. 11-29). Montréal: Presses de l’Université de Montréal.

42.

Roche,

(2012). Ontologie : entre terminologie et connaissance de spécialité [conference paper]. Paper presented at the Conference GLAT – Terminologies: textes, discours et accès aux savoirs spécialisés, Genova, Italy. Available from https://hal.archives-ouvertes.fr/hal-01180282/file/Roche-GLAT%202012.pdf.

43.

Sayer,

(2000). Realism and social science. London: Sage.

44.

Schwandt,

(2007). Dictionary of qualitative inquiry (3rd ed.). Thousand Oaks: Sage.

45.

Surowiecki,

(2007). The wisdom of crowds. New York: Random House.

46.

Tang,

Pluye,

, & Bouthillier,

(2015). eSRAP: système de surveillance de tendances en recherche permettant la collaboration des membres de communautés en recherche axée sur le patient (RAP) pour évaluer et partager les résultats et les projets de recherche de maniére adaptée [Registration of Copyright # 1126124]. Canadian Intellectual Property Office, Industry Canada. Ottawa.

47.

Tashakkori,

, & Teddlie,

(2010). Handbook of mixed methods in social and behavioral research. Thousand Oaks: Sage.

48.

Teddlie,

, & Tashakkori,

(2003). Major issues and controversies in the use of mixed methods in the social and behavioural sciences. In Tashakkori

& Teddlie

(Eds.), Handbook of mixed methods in social and behavioral research (pp. 3-50). Thousand Oaks: Sage.

49.

Teddlie,

, & Tashakkori,

(2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Thousand Oaks: Sage.

50.

Teddlie,

, & Tashakkori,

(2010). Overview of contemporary issues in mixed methods research. In Tashakkori

& Teddlie

(Eds.), Handbook of mixed methods in social and behavioral research (pp. 1-41). Thousand Oaks: Sage.

51.

Thomas,

Noel-Storr,

Marshall,

Wallace,

McDonald,

Mavergames,

, et al. Living Systematic Review Network. (2017). Living systematic reviews: 2. Combining human and machine effort. Journal of Clinical Epidemiology,91, 31-37.

52.

Tyson,

(2014). Critical theory today. New York: Routledge.

53.

Van de Ven,

A. H.

(1992). Suggestions for studying strategy process: a research note. Strategic Management Journal,13(5), 169-188.

54.

Wazny,

(2017). “Crowdsourcing” ten years in: A review. Journal of Global Health,7(2), 1-15.

55.

Weiss,

(2016). Crowdsourcing literature reviews in new domains. Technology Innovation Management Review,6(2), 5-14.

56.

Yin,

(2006). Mixed methods research: Are the methods genuinely integrated or merely parallel?Research in the Schools,13(1), 41-47.

57.

Yin,

(2014). Case study research: Design and methods. Thousand Oaks: Sage.

58.

Youngs,

, & Piggot-Irvine,

(2011). The application of a multiphase triangulation approach to mixed methods: The research of an aspiring school principal development program. Journal of Mixed Methods Research,6(3), 184-198.

Facilitators and barriers associated with the implementation of an innovative cross-disciplinary monitoring of the scientific literature: The Collaborative eBibliography on Mixed Methods (CeBoMM) – A research protocol

Abstract

Keywords

1. Introduction

2. Rationale

2.1 Pluralism of worldviews in MM

2.2 Common types of MM designs

2.3 Three types of integration in MM

2.3.1 Connection of phases

2.3.2 Comparison of results

2.3.3 Assimilation of data

2.4 The value of collaborative monitoring of scientific literature

3.1 Phase 1

3.2 Phase 2

4. Expected results and knowledge translation

Footnotes

Acknowledgments

References