Evaluating online health information sources using a mixed methods approach. Part 3

Abstract

1. Introduction

Part 1 of the special issue on “Evaluating online health information sources using a mixed methods approach” (Granikov & Pluye 2018, volume 34(4) of the journal) focused on the role of research evidence, literature reviews, their history, specific methods and tools. In Part 2, authors shared findings related to evaluating online health information, innovative tools and health information literacy interventions (Granikov & Pluye, 2019, volume 35(1) of the journal). This third and final part presents more innovative ideas, research protocols and results from members of the Information Technology Primary Care Research Group (ITPCRG) from the Department of Family Medicine at McGill University in Montreal, Canada. Being guest editors for this issue but also active members of this group, we acknowledge that we co-authored some articles written in these series.

The ITPCRG is transdisciplinary, with members exchanging expertise in information, computer, and health sciences, to plan and carry out research projects. In line with the journal’s interest, the common thread uniting ITPCRG projects is information, information practices as topics of research, teaching, and learning in health sciences. Specifically, the articles selected here highlight the following two research areas: the role of information from the viewpoint of individuals, and collective intelligence and health information.

The articles in the first research area address the role and value of information and information practices for learning, practice, and teaching in health-related domains, be it for practicing clinicians, residents, medical students, managers, researchers, patients and information consumers. Many of the studies reported are based on data collected with the Information Assessment Method (IAM) from different user groups. The IAM allows to systematically document the impact of information and stimulate reflection that contributes to learning (www.mcgill.ca/iam). The article by Granikov et al. introduces the tool and provides an overview of its development and implementation.

Information from research-based results and related information practices are at the heart of the evidence-based movement, such as evidence-informed clinical practice, public health and knowledge translation (i.e., program diffusion, dissemination to target audience, implementation in practice, sustainability and scaling up). In other words, practitioners need to be aware and consequently use the latest reliable research-based evidence to ensure that the right patient receives the right treatment at the right time. Unfortunately, practitioners continue to face barriers when it comes to implementing evidence in practice. The article by Asfour et al. reports on using the IAM in the context of clinical practice guidelines (sets of recommendations) for cancer survivors and investigates why such guidelines are not always implemented. Kluchnyk et al. discuss the implementation of the IAM in the context of spaced-education and online learning of medical residents. Gonzalez-Reyes et al. share findings from an IAM-based project in a continuing medical education program for family physicians and pharmacists, observing that participants who expected health benefits for their patients following an e-learning activity, were more likely to participate in this program in the future. Closing these research papers on information and education is the article by Frati, who shares her experience as an academic librarian applying a problem-based approach to teach evidence-informed practice in nursing.

The articles in the second research area go beyond individual information and behaviour, traditionally investigated in Library and Information Science (LIS) research, and study the effects of collective information (i.e., collective ways of creating, finding and using information). We all use collective information (Bates, 2010). The most frequently cited researchers in the field have studied collective information in everyday life and at the workplace (Chatman, 1996; Dervin, 2003; Dervin & Nilan, 1986; Fisher et al., 2005; Pettigrew et al., 2001; Savolainen, 2002, 2005, 2007). Specifically, their research results suggest that collective information facilitates the interpretation and use of complex informational content. Web developments put collective information at the heart of contemporary social actions (Burris, 1998; DiMaggio et al., 2001).

Collective information has three positive effects: decision support, learning and problem identification; and a negative effect of unnecessary content overload (Farmer, 2016). Collective information is associated with the expansion of Web 2.0 that created new forms of collective intelligence (CI) such as the continuous co-construction of knowledge with the population (e.g., wikis) (Jimenez-Diaz et al., 2018).

CI is immemorial and consists of sparks of wisdom produced by a group (Malone & Bernstein, 2015; Surowiecki, 2005). It is useful for managing novelty by providing plausibility, consistency and practical reason; and it can produce informational content and collective information (De Liddo et al., 2012). Mobilizing CI of groups through crowdsourcing is an emergent trend proposing new types and methods for research across disciplines (Nguyen et al., 2019). CI has been used to create ‘intellectual outputs’, generate ideas, evaluate ideas or work, and solve problems (Nguyen et al., 2019).

The article by Granikov et al. illustrates CI and collaborative information by way of assessing a collaborative system to monitor and filter research publications, called eSRAP. eSRAP is an innovative platform created by ITPCRG with the support of the National Research Council of Canada and the Quebec SPOR SUPPORT Unit (Tang et al., 2015). eSRAP has been designed to allow communities of clinicians, health managers, patient partners, and researchers to collaborate in monitoring and filtering the literature on topics of interest. This type of CI and collective information refers to a complex socio-technical autopoietic self-referential system (Adams et al., 2005; Luhmann, 1995) that uses the knowledge of Internet users to create informational content at low cost (e.g., Wikipedia) (Lichten et al., 2018).

Such a system creates a way to learn by cooperating with others. It is associated with positive effects (e.g., saving time and money, complete and up-to-date content) and a negative effect related to uncertainty about information quality (Brown & Allison, 2014; Cox et al., 2015; Francq, 2011; Morris & Teevan, 2010). From the viewpoint of information professionals, educators, and students, knowing a novel solution for keeping up to date by continuously monitoring and filtering the literature is highly relevant. In addition, this paper can help all those who wish to implement a community-based collaborative surveillance of the literature in health and social sciences.

The article by Pluye et al. describes how eSRAP could be incorporated in the conception, development and sustainability of online academic reference books. Using mixed methods as an example, their project is aimed to identify and measure the importance of facilitators and barriers associated with the implementation of the Collaborative eBibliography on Mixed Methods (CeBoMM). Ultimately, this can be adapted to be used by teachers in other academic areas, and those interested in collaborative information monitoring of a topic. This initiative may inform the creation of other Collaborative eBibliographies (CeBoTOPIC-NAME). In this paper, Mixed Methods (MM) involve combining qualitative (QUAL) and quantitative (QUAN) methods in program evaluation, primary research, and literature reviews. MM are being increasingly used in health and social sciences. CeBoMM can find current illustrative examples of MM strategies for trainees and identify emerging innovative strategies. This work is important given that MM is increasingly popular in LIS and is of interest for graduate students and researchers.

The article by Grad and Tang is based on a 13-year longitudinal survey involving thousands of Canadian physicians and linking the Information Assessment Method (IAM) to educational emails containing research-based synopses relevant for patients’ health, called POEMs (Patient Oriented Evidence that Matters). In the context of continuous education programs, authors use IAM to collect physicians’ constructive feedback comments (experiential information) about scientific knowledge embedded in POEMs. For example, a POEM is emailed daily to members of the Canadian Medical Association, who can then rate and comment on it using IAM (Grad et al., 2014). Consequently, web editors can use such comments to improve their informational content (Tang et al., 2015). In the present paper, these comments are seen as collective intelligence that creates collaborative experiential information for improving the IAM-POEM-based continuing education program. The comments can contribute to physicians’ knowledge and improve health care and patient health. Considering that the IAM is being used worldwide in more than 20 contexts, this paper can benefit information professionals in multiple fields, who seek to improve their use of evidence-based abstracts of scientific publications and experience-based information (information users’ comments) in their daily work.

The article by El Sherif et al concludes this special issue by describing a free online application called ATCER (Automated Text Classifier of Empirical Research). ATCER is based on the results of two complementary LIS-oriented studies that aimed to (1) establish a new bibliographic search filter, called the mixed filter (enabling the retrieval of empirical studies) and (2) compare the performance of machine learning solutions that automatize the mixed filter. The most performant solution was integrated in ATCER and can be used by graduate students, managers, practitioners, and researchers in health and social sciences to analyze their bibliographies and categorize records into “empirical” and “nonempirical”.

Knowing about this automated text classification system is important for information researchers and practitioners. Methodological reviews with descriptive analysis of empirical studies are common in LIS (Chu, 2015). Specifically, the authors, may analyse all types of empirical studies published in core LIS journals to describe the use and prevalence of various research methods (Fidel, 2008; Julien et al., 2011). ATCER can also be employed by librarians developing search strategies for literature reviews or to recommend to researchers and students working on mixed studies reviews (reviews including empirical studies with diverse designs: qualitative, quantitative and mixed methods designs).

In conclusion, there is much to gain from research on the IAM, CI, and collaborative information in health and social sciences. The IAM being a validated tool to systematically assess the impact of health information from the viewpoints of different user groups, could be used by information researchers and professionals in information evaluation projects. Mobilizing collective intelligence (CI) of groups through crowdsourcing is an emergent trend proposing new types and methods for research across disciplines (Nguyen et al., 2019). CI has been used to create ‘intellectual outputs’, generate ideas, evaluate ideas or work, and solve problems (Nguyen et al., 2019). The articles in this special issue report protocols and projects pertaining to health information research. Their methodology and results are nevertheless transferable and potentially valuable to LIS educators, professionals, graduate students, and researchers.

References

Adams,

Blandford,

Budd,

, & Bailey,

(2005). Organizational communication and awareness: A novel solution for health informatics. Health Informatics Journal, 11(3), 163-178.

Bates,

(2010). Information behavior. In Bates

& Niles Maack

, (Eds.), Encyclopedia of library and information sciences (3rd ed., Vol. 3, pp. 2381-2391). New York: CRC Press.

Brown,

, & Allison,

(2014). Using crowdsourcing to evaluate published scientific literature: Methods and example. PLoS One, 9(7), e100647.

Burris,

(1998). Computerization of the workplace. Annual Review of Sociology, 24(1), 141-157. aaa(000) Chatman, E. (1996). The impoverished life-world of outsiders. Journal of the American Society for Information Science and Technology, 47(3), 193-206. aaa(000) Chu, H. (2015). Research methods in library and information science: A content analysis. Library & Information Science Research, 37(1), 36-41.

Cox,

Oh,

Simmons,

Lintott,

Masters,

Greenhill,

, et al. (2015). Defining and measuring success in online citizen science: A case study of zooniverse projects. Computing in Science & Engineering, 17(4), 28-41.

De Liddo,

Sńdor,

Á.

, & Shum,

(2012). Contested collective intelligence: Rationale, technologies, and a human-machine annotation study. Computer Supported Cooperative Work, 21(4-5), 417-448.

Dervin,

(2003). Human studies and user studies: A call for methodological inter-disciplinarity. Information Research, 9(1), 1-16. Retrieved from Information Research website: http://informationr.net/ir/9-1/paper166.html.

Dervin,

, & Nilan,

(1986). Information needs and uses. Annual Review of Information Science and Technology, 21, 3-33.

DiMaggio,

Hargittai,

Neuman,

, & Robinson,

(2001). Social implications of the internet. Annual Review of Sociology, 27(1), 307-336.

10.

Farmer,

(2016). Using virtual environments to transform collective intelligence. In Analyzing digital discourse and human behavior in modern virtual environments (pp. 149-163). Hershey, PA: IGI Global.

11.

Fidel,

(2008). Are we there yet?: Mixed methods research in library and information science. Library & Information Science Research, 30(4), 265-272.

12.

Fisher,

Erdelez,

, & McKechnie,

(2005). Theories of information behavior. Medford, NJ: American Society for Information Science and Technology by Information Today.

13.

Francq,

(2011). Collaborative search and communities of interest: Trends in knowledge sharing and assessment. Hershey, PA: Information Science Reference.

14.

Grad,

Pluye,

Shulha,

Tang,

Moscovici,

Repchinsky,

, & Meuser,

(2014). Big data from the push of clinical information: Harvesting user feedback for continuing education. In Marconi

& Lehmann

(Eds.), Big data and health analytics (pp. 79-104). New York: Taylor and Francis.

15.

Jimenez-Diaz,

Serrano,

, & Arroyo,

(2018). A wikia census: Motives, tools and insights. Paper presented at the Proceedings of the 14th International Symposium on Open Collaboration.

16.

Julien,

Pecoskie,

J. J.

, & Reed,

(2011). Trends in information behavior research, 1999–2008: A content analysis. Library & Information Science Research, 33(1), 19-24.

17.

Lichten,

Ioppolo,

d’Angelo,

Simmons,

, & Morgan Jones,

(2018). Citizen science: Crowdsourcing for research. Cambridge, UK: The Healthcare Improvement Studies Institute.

18.

Luhmann,

(1995). Social systems. Stanford, CA: Stanford University Press.

19.

Malone,

, & Bernstein,

(2015). Handbook of collective intelligence. Cambridge, MA: MIT Press.

20.

Morris,

, & Teevan,

(2010). Collaborative web search: Who, what, where, when, and why. San Rafael, CA: Morgan & Claypool Publishers.

21.

Nguyen,

V. T.

Benchoufi,

Ghosn,

Ravaud,

, & Boutron,

(2019). A scoping review provided a framework for new ways of doing research through mobilizing collective intelligence. Journal of Clinical Epidemiology. 110, 1-11.

22.

Pettigrew,

Fidel,

, & Bruce,

(2001). Conceptual frameworks in information behavior. Annual Review of Information Science and Technology, 35, 43-78.

23.

Savolainen,

(2002). Network competence and information seeking on the internet: From definitions towards a social cognitive model. Journal of Documentation, 58(2), 211-226.

24.

Savolainen,

(2005). Everyday life information seeking. In Fisher

Erdelez

, & McKechnie

(Eds.), Theories of information behavior (pp. 143-148). Medford, NJ: Information Today Inc.

25.

Savolainen,

(2007). Information behavior and information practice. The Library Quarterly, 77(2), 109-132.

26.

Surowiecki,

(2005). The wisdom of crowds. New York: Anchor Books.

27.

Tang,

Bouthillier,

Pluye,

Grad,

, & Repchinsky,

(2015). The value of user feedback: Healthcare professionals’ comments to the health information provider. Journal of the Association for Information Science and Technology, 66(2), 377-391.

28.

Tang,

Pluye,

, & Bouthillier,

(2015). eSRAP: Système de surveillance de tendances en recherche permettant la collaboration des membres de communautés en recherche axée sur le patient (rap) pour évaluer et partager les résultats et les projets de recherche de manière adaptée (Registration of copyright # 1126124).