Purpose

To quantify how large language model (LLM) assistance influences otolaryngology residents’ operative planning in a simulation-based setting.

Methods

In a within-subjects paired design, 13 otolaryngology residents (PGY1-4) completed 21 synthetic surgical scenarios twice: first independently and then after reviewing a ChatGPT-5–generated plan produced using a fixed prompt. After AI review, residents recorded an ordinal ImpactScore (0-3: no change to major change) describing the magnitude of plan modification and rated usefulness (1-5 Likert). Paired analyses compared pre- vs post-AI changes in plan structure and narrative length across domains; mixed-effects models evaluated associations with PGY level and subspecialty.

Results

Across all 267 paired resident–case observations, AI altered operative plans in 91.0% (243/267; 95% CI, 87.2%–93.8%), with a mean ImpactScore of 1.32 ± 0.75 (median 1 [IQR 1-2]). Impact differed by PGY (H = 25.02; P = 1.53 × 10⁻⁵), with PGY2 and PGY3 showing lower ImpactScores than PGY1 in mixed-effects modeling. Subspecialty differences were observed (H = 11.95; P = 0.036), with higher impact in sleep surgery than laryngology. After AI review, operative step descriptions shortened (mean −192.2 characters; P < 10⁻⁴⁰) and safety text shortened (−18.3 characters; P < 10⁻²²), while complication text modestly increased (+8.1 characters; P = 0.0039). Mean usefulness was 2.90 ± 1.10 and increased with ImpactScore (β = 0.264; P = 0.003).

Conclusion

In simulation-based operative planning, ChatGPT-5 frequently prompted residents to modify and reorganize operative plans, with effects varying by training level and case type. LLM assistance may function as a reflective structuring aid, but requires supervised use given occasional clinically concerning model suggestions.