Pediatric Nurses’ Perceptions of the Quality and Usefulness of Clinical Summaries: Surgeons Versus Generative AI

Abstract

Purpose: The purpose of this study was to compare the performance of artificial intelligence (AI)-generated and physician-written handoff summaries in a pediatric surgery population from nurses’ perspective, utilizing a validated quality assessment tool. Methods: Three clinical nurses independently reviewed and compared AI-generated summaries to physician-written summaries for patients admitted to the surgery service line utilizing the validated Physician Documentation Quality Instrument (PDQI-9) tool. The nurse evaluation team met prior to scoring to ensure consistency of interpretation and scoring of PDQI-9 domains. Results: Artificial intelligence-generated and physician written summaries for 117 patients were included, revealing a statistically significant higher ranking for AI-generated summaries (Z = 3.89, p < .001). Artificial intelligence-generated summaries outperformed physician written summaries in seven out of nine PDQI-9 domains (thorough, useful, organized, comprehensible, succinct, synthesized, and internally consistent). Fleiss’ Kappa analysis was conducted to assess the degree of agreement among the raters. The results showed fair agreement for five domains and slight agreement for four domains, highlighting the need and complexity of multirater review of clinical summaries. Practice Implications/Conclusion: This is the first study to examine how nurses evaluate the quality and usefulness of AI-generated summaries in their practice. Artificial intelligence-generated physician summaries improve the quality of handoff for nurses by providing clear and concise information, reducing variability, and providing structure for information sharing. Integration into practice may improve communication, enhance nursing workflow, and improve patient safety. Further research is needed to evaluate AI-generated physician summaries’ long-term effectiveness in patient safety, workload efficiency, and nursing satisfaction.

Keywords

clinical documentation generative artificial intelligence pediatric surgery nurse

Get full access to this article

View all access options for this article.

References

Amarasingham

Swanson

T. S.

Treichler

D. B.

Amarasingham

S. N.

Reed

W. G.

(2010). A rapid admission protocol to reduce emergency department boarding times. BMJ Quality & Safety, 19(3), 200–204. https://doi.org/10.1136/qshc.2008.028340

Arsoniadis

E. G.

Olsen

R. K.

Skube

Marquard

Melton

G. B.

(2019). Resident-physician preferences for electronic handoff note content: Implications for implementation of a system-wide electronic health record-integrated handoff tool. Studies in Health Technology and Informatics, 264, 1121–1125. https://doi.org/10.3233/SHTI190400

Barak-Corren

Wolf

Rozenblum

Creedon

J. K.

Lipsett

S. C.

Lyons

T. W.

Michelson

K. A.

Miller

K. A.

Shapiro

D. J.

Reis

B. Y.

Fine

A. M.

(2024). Harnessing the power of generative AI for clinical summaries: Perspectives from emergency physicians. Annals of Emergency Medicine, 84(2), 128–138. https://doi.org/10.1016/j.annemergmed.2024.01.039

Carroll

J. S.

Williams

Gallivan

T. M.

(2012). The ins and outs of change of shift handoffs between nurses: A communication challenge. BMJ Quality & Safety, 21(7), 586–593. https://doi.org/10.1136/bmjqs-2011-000614

Cohen

M. D.

Hilligoss

P. B.

(2010). The published literature on handoffs in hospitals: Deficiencies identified in an extensive review. BMJ Quality & Safety, 19(6), 493–497. https://doi.org/10.1136/qshc.2009.033480

Davis

F. D.

(1989). Technology acceptance model: TAM. In Al-Suqri

M. N.

Al-Aufi

A. S.

(Eds.), Information seeking behavior and technology adoption (pp. 205–219). IGI Global.

Decker

Trang

Ramirez

Colley

Pierce

Coleman

Bongiovanni

Melton

G. B.

Wick

(2023). Large language model-based chatbot vs surgeon-generated informed consent documentation for common procedures. JAMA Network Open, 6(10), e2336997. https://doi.org/10.1001/jamanetworkopen.2023.36997

Desmedt

Ulenaers

Grosemans

Hellings

Bergs

(2021). Clinical handover and handoff in healthcare: A systematic review of systematic reviews. International Journal for Quality in Health Care, 33(1), mzaa170. https://doi.org/10.1093/intqhc/mzaa170

Galatzan

B. J.

Johnson

Judson

Shan

(2024). Linguistic dissection of nursing handoffs: Implications for patient safety in varied-acuity hospital settings. Journal of Clinical Nursing (John Wiley & Sons, Inc.), 33(8), 3077–3088. https://doi.org/10.1111/jocn.17190

10.

Ghonem

N. M. E.

El-Husany

W. A.

(2023). Handoff communication practices among nurses: A cross-sectional study. SAGE Open Nursing, 9, 23779608231159340. https://doi.org/10.1177/23779608231159340

11.

Harris

P. A.

Taylor

Minor

B. L.

Elliott

Fernandez

O’Neal

McLeod

Delacqua

Kirby

Duda

S. N.

& REDCap Consortium . (2019). The REDCap consortium: Building an international community of software partners. Journal of Biomedical Informatics, 95, 103208. https://doi.org/10.1016/j.jbi.2019.103208

12.

Harris

P. A.

Taylor

Thielke

Payne

Gonzalez

Conde

J. G.

(2009). Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42(2), 377–381. https://doi.org/10.1016/j.jbi.2008.08.010

13.

Keenan

Yakel

Dunn Lopez

Tschannen

Ford

Y. B.

(2013). Challenges to nurses’ efforts of retrieving, documenting, and communicating patient care information. Journal of the American Medical Informatics Association, 20(2), 245–251. https://doi.org/10.1136/amiajnl-2012-001054

14.

Mamalelala

T. T.

Holzemer

W. L.

Seloilwe

E. S.

Iwu

(2025). Experiences and challenges of rural nurses in Botswana handling births outside the hospital environment. Journal of Nursing Scholarship, 57(4), 695–702. https://doi.org/10.1111/jnu.70015

15.

Miller

(2021). I-PASS as a nursing communication tool. Pediatric Nursing, 47(1), 30–37. https://doi.org/10.62116/pnj.2021.47.1.30

16.

O'Rourke

Abraham

Riesenberg

L. A.

Matson

Lopez

K. D.

(2018). A Delphi study to identify the core components of nurse to nurse handoff. Journal of Advanced Nursing, 74(7), 1659–1671. https://doi.org/10.1111/jan.13565

17.

Patton

L. J.

Tidwell

J. D.

Falder-Saeed

K. L.

Young

V. B.

Lewis

B. D.

Binder

J. F.

(2017). Ensuring safe transfer of pediatric patients: A quality improvement project to standardize handoff communication. Journal of Pediatric Nursing, 34, 44–52. https://doi.org/10.1016/j.pedn.2017.01.004

18.

Rhudy

L. M.

Johnson

M. R.

Krecke

C. A.

Keigley

D. S.

Kraft

S. J.

Maxson

P. M.

McGill

S. M.

Warfield

K. T.

(2022). Standardized change-of-shift handoff: Nurses’ perspectives and implications for evidence-based practice. American Journal of Critical Care, 31(3), 181–188. https://doi.org/10.4037/ajcc2022629

19.

Schuster

Jenq

Thung

Hersh

Nunes

Silverman

Horwitz

(2014). Electronic handoff instruments. Journal of the American Medical Informatics Association, 21(e2), e352–e357. https://doi.org/10.1136/amiajnl-2013-002361

20.

Shinners

Aggar

Grace

Smith

(2020). Exploring healthcare professionals’ understanding and experiences of artificial intelligence technology use in the delivery of healthcare: An integrative review. Health Informatics Journal, 26(2), 1225–1236. https://doi.org/10.1177/1460458219874641

21.

Stetson

P. D.

Bakken

Wrenn

J. O.

Siegler

E. L.

(2012). Assessing electronic note quality using the physician documentation quality instrument (PDQI-9). Applied Clinical Informatics, 3(2), 164–174. https://doi.org/10.4338/ACI-2011-11-RA-0070

22.

Walia

Qayumi

Khawar

Dygulska

Bialik

Salafia

Narula

(2016). Physician transition of care: Benefits of I-PASS and an electronic handoff system in a community pediatric residency program. Academic Pediatrics, 16(6), 519–523. https://doi.org/10.1016/j.acap.2016.03.005