A pipeline for medical literature search and its evaluation

Abstract

One database commonly used by clinicians for searching the medical literature and practicing evidence-based medicine is PubMed. As the literature grows, it has become challenging for users to find the relevant material quickly because most of the time the relevant results are not on the top. In this article, we propose a search and ranking pipeline to improve the search results based on relevancy. We first propose an ensemble model consisting of three classifiers: bidirectional long-short-term memory conditional random field (bi-LSTM-CRF), support vector machine and naive Bayes to recognise PICO (patient, intervention, comparison, outcome) elements from abstracts. The ensemble was trained on an annotated corpus consisting of 5000 abstracts split into 4000 and 1000 training and testing data, respectively. The ensemble recorded an accuracy of 93%. We then retrieved around 927,000 articles from PubMed for the years 2017–2021 (access date 16 April 2021). For every abstract, we extracted and grouped all P, I and O terms, and stored these groups along with the article ID in a separate database. During the search, every P, I and O term of the query is searched only in its corresponding group of every abstract. The scoring method simply counts the number of matches between the query’s P, I and O elements and the words in P, I and O groups, respectively. The abstracts are sorted by the number of matches and the top five abstracts are listed using their pre-stored abstract IDs. A comprehensive user study was conducted where 60 different queries were formulated and used to generate ranked search results using both PubMed and our proposed model. Five medical professionals assessed the ranked search results and marked every item as relevant or non-relevant. Both models were compared using precision@K and mean-average-precision@K metrics where K is 5. For most of the queries, our model produced higher precision@K values than PubMed. The mean-average-precision@K value of our model is also higher than PubMed (0.83 versus 0.67).

Keywords

Bi-LSTM-CRF ensemble MongoDB PICO framework PubMed search strategy

Get full access to this article

View all access options for this article.

References

The BMJ. Poll results, 2021, https://www.bmj.com/content/suppl/2007/01/18/334.suppl_1.DC3

PubMed.org. PubMed, 2021, https://pubmed.ncbi.nlm.nih.gov/ (accessed 16 March 2021).

Mao

. Mesh now: automatic mesh indexing at PubMed scale via learning to rank. J Biomed Semantics 2017; 8(1): 15.

Kari

. Correlation between lack of information documentation and underdevelopment in Nigeria: empirical evidence from experts. Ianna J Interdiscip Stud 2022; 4(1): 35–43, https://www.iannajournalofinterdisciplinarystudies.com/index.php/1/article/view/84

Westbrook

Coiera

Gosling

. Do online information retrieval systems help experienced clinicians answer clinical questions? J Am Med Inf Assoc 2005; 12(3): 315–321.

Salvador-Oliván

Marco-Cuenca

Arquero-Avilés

. Errors in search strategies used in systematic reviews and their effects on information retrieval. J Med Libr Assoc 2019; 107(2): 210–221.

Kumar

Sharma

. Information retrieval system: an overview, issues, and challenges. Int J Technol Diffus 2018; 9(1): 1–10.

Turner

. Evidence-based practice in health: introduction, 2014, https://canberra.libguides.com/c.php?g=599346

Liew

et al. Development of a search strategy for an evidence based retrieval service. PLoS ONE 2016; 11(12): e0167170.

10.

Schardt

Adams

Owens

et al. Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Med Inf Decis Mak 2007; 7(1): 16.

11.

Dimsdale

Carlson

. Teaching with librarians: using APA PsycINFO and the PICO framework in education instruction, 2020, https://scholarworks.gsu.edu/univ_lib_facpres/39/

12.

Huang

Chiang

Xiao

et al. PICO element detection in medical text without metadata: are first sentences enough? J Biomed Inf 2013; 46(5): 940–946.

13.

Jin

Szolovits

. Advancing PICO element detection in biomedical text via deep neural networks. Bioinformatics 2020; 36: 3856–3862.

14.

Wallace

Kuiper

Sharma

et al. Extracting PICO sentences from clinical trial reports using supervised distant supervision. J Mach Learn Res 2016; 17(1): 4572–4596.

15.

Boudin

Nie

Bartlett

et al. Combining classifiers for robust PICO element detection. BMC Med Inf Decis Mak 2010; 10(1): 29.

16.

Zhang

Geng

Zhang

et al. Aceso: PICO-guided evidence summarization on medical literature. IEEE J Biomed Health Inform 2020; 24(9): 2663–2670.

17.

Patel

Yang

Marshall

et al. Syntactic patterns improve information extraction for medical search. Proc conf 2018; 2018: 371–377.

18.

Chabou

Iglewski

. Combination of conditional random field with a rule based method in the extraction of PICO elements. BMC Med Inf Decis Mak 2018; 18(1): 1–14.

19.

Jin

Szolovits

. PICO element detection in medical text via long short-term memory neural networks. In: Proceedings of the BioNLP 2018 workshop, pp. 67–75. Association for Computational Linguistics. DOI: 10.18653/v1/W18-2308.

20.

Stylianou

Vlahavas

. Transformed: end-to-end transformers for evidence-based medicine and argument mining in medical literature. J Biomed Inf 2021; 117: 103767.

21.

Liu

Sun

et al. Sent2span: span detection for PICO extraction in the biomedical text without span annotations. In: Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1705–1715. Association for Computational Linguistics. DOI: 10.18653/v1/2021.findings-emnlp.147.

22.

Kloda

Bartlett

. A characterization of clinical questions asked by rehabilitation therapists. J Med Libr Assoc 2014; 102(2): 69–77.

23.

Kloda

Boruff

Cavalcante

. A comparison of patient, intervention, comparison, outcome (PICO) to a new, alternative clinical question framework for search skills, search results, and self-efficacy: a randomized controlled trial. J Med Libr Assoc 2020; 108(2): 185–194.

24.

Eriksen

Frandsen

. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review. J Med Libr Assoc 2018; 106(4): 420–431.

25.

Liu

Anstey

et al. Rethinking PICO in the machine learning era: Ml-PICO. App Clin Inf 2021; 12(2): 407–416.

26.

Cooke

Smith

Booth

. Beyond PICO: the spider tool for qualitative evidence synthesis. Qual Health Res 2012; 22(10): 1435–1443.

27.

Methley

Campbell

Chew-Graham

et al. PICO, PICOs and spider: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res 2014; 14(1): 1–10.

28.

Rehman

Campbell

Chew-Graham

et al. Difference between quantitative and qualitative research question-PICO vs. spider. Am Sci Res J Eng Technol Sci 2021; 77(1): 188–199.

29.

Bethesda

. Entrez programming utilities help. Bethesda, MD: National Center for Biotechnology Information, 2010.

30.

Chalapathy

Borzeshi

Piccardi

. Bidirectional LSTM-CRF for clinical concept extraction. In: Proceedings of the clinical natural language processing workshop (ClinicalNLP), 13–16 December 2016, pp. 7–12. Osaka International Convention Center, Osaka, Japan: The COLING 2016 Organizing Committee.

31.

Jie

. Dependency-guided LSTM-CRF for named entity recognition. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp. 3862–3872. Association for Computational Linguistics. DOI: 10.18653/v1/D19-1399.

32.

Qin

Zeng

. Research of clinical named entity recognition based on bi-LSTM-CRF. J Shanghai Jiaotong Univ (Sci) 2018; 23(3): 392–397.

33.

Nye

Patel

et al. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. Proc Conf Assoc Comput Linguist Meet 2018; 2018: 197–207.

34.

Brockmeier

Przybyła

et al. Improving reference prioritisation with PICO recognition. BMC Med Inf Decis Mak 2019; 19(1): 1–14.

35.

Kang

Perotte

Tang

et al. UMLS-based data augmentation for natural language processing of clinical research literature. J Am Med Inf Assoc 2021; 28(4): 812–823.