Guideline-Integrated Large Language Models Improve Decision Support for Acute Ear, Nose and Throat Emergencies.

BACKGROUND

Acute otolaryngologic emergencies such as sudden sensorineural hearing loss (SSNHL), epistaxis, and acute facial paralysis frequently present to emergency departments (EDs), where early guideline-concordant decisions are critical to prevent irreversible morbidity. Large language models (LLMs) are increasingly explored for clinical decision support, yet their ability to consistently apply specialty guidelines in time-sensitive settings remains uncertain. Retrieval-augmented generation (RAG), which supplies LLMs with authoritative guideline content during response generation, may improve accuracy and safety.

OBJECTIVES

To determine whether integrating American Academy of Otolaryngology-Head and Neck Surgery (AAOHNS) Clinical Practice Guidelines via RAG improves the accuracy, guideline adherence, and safety of LLM-generated decision support for acute ear, nose and throat (ENT) emergencies in the ED.

METHODS

Twelve standardized clinical vignettes representing ED presentations of SSNHL, epistaxis, and Bell's palsy were developed. Two LLMs (ChatGPT and Gemini) generated responses under baseline conditions and with guideline-integrated RAG. Five blinded otolaryngologists independently evaluated outputs using a nine-domain rubric assessing diagnostic accuracy, diagnostic workup, management quality, referral appropriateness, guideline adherence, and risk of misleading information. Non-parametric tests compared performance, and inter-rater reliability was assessed using intraclass correlation coefficients.

RESULTS

Inter-rater reliability was excellent. Compared with baseline models, RAG-enabled models demonstrated significantly improved diagnostic accuracy, diagnostic workup quality, management planning, referral appropriateness, and guideline adherence. RAG reduced misleading recommendations and modestly decreased overtreatment, while overtesting rates were unchanged.

CONCLUSIONS

Guideline integration through RAG improves the reliability, safety, and guideline alignment of LLM outputs for acute otolaryngologic emergencies and may support evidence-based decision-making in emergency care.