A modular pipeline for natural language processing-screened human abstraction of a pragmatic trial outcome from electronic health records

Sheikhalishahi S Miotto R Dudley JT , et al. Natural language processing of clinical notes on chronic diseases: systematic review. JMIR Med Inform 2019; 7(2): e12239.

Spasic I Nenadic G. Clinical text data in machine learning: systematic review. JMIR Med Inform 2020; 8(3): e17984.

Lee RY Kross EK Downey L , et al. Efficacy of a communication-priming intervention on documented goals-of-care discussions in hospitalized patients with serious illness: a randomized clinical trial. JAMA Netw Open 2022; 5(4): e225088.

Curtis JR Lee RY Brumback LC , et al. Improving communication about goals of care for hospitalized patients with serious illness: Study protocol for two complementary randomized trials. Contemp Clin Trials 2022; 120: 106879.

Curtis JR Lee RY Brumback LC , et al. Intervention to promote communication about goals of care for hospitalized patients with serious illness: a randomized clinical trial. JAMA 2023; 329(23): 2028–2037.

Secunda K Wirpsa MJ Neely KJ , et al. Use and meaning of “goals of care” in the healthcare literature: a systematic review and qualitative discourse analysis. J Gen Intern Med 2019; 35(5): 1559–1566.

Edmonds KP Ajayi TA. Do we know what we mean? An examination of the use of the phrase “goals of care” in the literature. J Palliat Med 2019; 22(12): 1546–1552.

Myers J Kim GL Bytautas J , et al. Differing conceptualizations of the goals of care discussion: a critical discourse analysis. J Pain Symptom Manage 2022; 63(4): 495–502.

Lee RY Brumback LC Lober WB , et al. Identifying goals of care conversations in the electronic health record using natural language processing and machine learning. J Pain Symptom Manage 2021; 61(1): 136–142.

Uyeda AM Curtis JR Engelberg RA , et al. Mixed-methods evaluation of three natural language processing modeling approaches for measuring documented goals-of-care discussions in the electronic health record. J Pain Symptom Manage 2022; 63(6): e713–e723.

Lee RY Kross EK Torrence J , et al. Assessment of natural language processing of electronic health records to measure goals-of-care discussions as a clinical trial outcome. JAMA Netw Open 2023; 6(3): e231204.

Lee RY Uyeda AM Li KS , et al. Performance of predefined search patterns for identifying documented goals-of-care discussions in inpatient electronic health records. J Palliat Med 2025; 28(7): 952–959.

Harris PA Taylor R Thielke R , et al. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42(2): 377–381.

Institute of Medicine. Dying in America: improving quality and honoring individual preferences near the end of life. Washington, DC: National Academies Press, 2015.

Teno JM Fisher ES Hamel MB , et al. Medical care inconsistent with patients’ treatment goals: association with 1-year Medicare resource use and survival. J Am Geriatr Soc 2002; 50(3): 496–500.

Mack JW Weeks JC Wright AA , et al. End-of-life discussions, goal attainment, and distress at the end of life: predictors and outcomes of receipt of care consistent with preferences. J Clin Oncol 2010; 28(7): 1203–1208.

Cosgriff JA Pisani M Bradley EH , et al. The association between treatment preferences and trajectories of care at the end-of-life. J Gen Intern Med 2007; 22(11): 1566–1571.

Turnbull AE Bosslet GT Kross EK. Aligning use of intensive care with patient values in the USA: past, present, and future. Lancet Respir Med 2019; 7(7): 626–638.

Bernacki RE Block SD and American College of Physicians High Value Care Task Force. Communication about serious illness care goals: a review and synthesis of best practices. JAMA Intern Med 2014; 174(12): 1994–2003.

Comer AR Hickman SE Slaven JE , et al. Assessment of discordance between surrogate care goals and medical treatment provided to older adults with serious illness. JAMA Netw Open 2020; 3(5): e205179.

Curtis JR Patrick DL Shannon SE , et al. The family conference as a focus to improve communication about end-of-life care in the intensive care unit: opportunities for improvement. Crit Care Med2001; 29(Suppl. 2): N26–N33.

Seaman JB Arnold RM Scheunemann LP , et al. An integrated framework for effective and efficient communication with families in the adult intensive care unit. Ann Am Thorac Soc 2017; 14(6): 1015–1020.

Dy SM Apostol C Martinez KA , et al. Continuity, coordination, and transitions of care for patients with serious and advanced illness: a systematic review of interventions. J Palliat Med 2013; 16(4): 436–445.

Sinuff T Dodek P You JJ , et al. Improving end-of-life communication and decision making: the development of a conceptual framework and quality indicators. J Pain Symptom Manage 2015; 49(6): 1070–1080.

You JJ Fowler RA Heyland DK , et al. Just ask: discussing goals of care with patients in hospital with serious illness. CMAJ 2014; 186(6): 425–432.

Jacobsen J Bernacki R Paladino J. Shifting to serious illness communication. JAMA 2022; 327(4): 321.

Chien I Shi A Chan A , et al. Identification of serious illness conversations in unstructured clinical notes using deep neural networks. In: Artificial Intelligence in Health: First International Workshop (ed Koch F Koster A Bichindaritz I , et al.), AIH 2018, Stockholm, 13–14 July 2018, pp.199–212. Revised Selected Papers. Lecture Notes in Artificial Intelligence. Cham: Springer.

Wilson CJ Newman J Tapper S , et al. Multiple locations of advance care planning documentation in an electronic health record: are they easy to find? J Palliat Med 2013; 16(9): 1089–1094.

Alsentzer E Murphy JR Boag W , et al. Publicly available clinical BERT embeddings. arXiv. https://arxiv.org/abs/1904.03323 (2019, accessed 29 August 2024).

Alsentzer E. BioClinicalBERT, 2019. https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT

Devlin J Chang MW Lee K , et al. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv. http://arxiv.org/abs/1810.04805 (2019, accessed 29 August 2024).

Udelsman B Chien I Ouchi K , et al. Needle in a haystack: natural language processing to identify serious illness. J Palliat Med 2019; 22(2): 179–182.

Lindvall C Deng CY Moseley E , et al. Natural language processing to identify advance care planning documentation in a multisite pragmatic clinical trial. J Pain Symptom Manage 2022; 63(1): e29–e36.

Vanderkam D. Local Turk. https://github.com/danvk/localturk (2024, accessed 22 January 2025)

Lee RY Li KS Sibley J , et al. NLP-screened EHR passage adjudication. REDCap Shared Library. January 22, 2025. https://redcap.vumc.org/consortium/library/search.php

Obeid JS McGraw CA Minor BL , et al. Procurement of shared data instruments for Research Electronic Data Capture (REDCap). J Biomed Inform 2013; 46(2): 259–265.

Manning C Surdeanu M Bauer J , et al. The Stanford CoreNLP natural language processing toolkit. In: Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, Baltimore, MD, June 2014, pp. 55–60. Association for Computational Linguistics.