Feasibility of Real-Time Integration of a Large Language Model in the EHR to Identify Goals-of-Care Documentation.

CONTEXT: This article describes integration of a large language model (LLM) into the electronic health record (EHR) to identify unstructured goals-of-care (GOC) documentation or discrete data entry.

OBJECTIVES

To determine whether a prompt-engineered LLM reliably detects GOC documentation in clinical notes comparable to human experts and can display this evaluation to clinicians in real time.

METHODS

An interdisciplinary team of palliative care experts created an adjudicated reference standard, which was used to evaluate LLM models in detecting GOC documentation. Given this, the best-fit model (GPT-4o) was then integrated with the EHR for a six-month pilot at four hospitals, then released enterprise-wide.

RESULTS

GPT-4o achieved the best performance (specificity 0.95, recall 0.74, precision 0.78). Manual audit on AI-flagged notes (n=325) confirmed substantial expert agreement (86%). In subgroup analyses, there was no statistically significant differences across race, sex, or age (specificity range 0.91-1; all p-values > 0.05). At enterprise scale, the model identified 37,000 patients with GOC documentation within approximately two months. In comparison to those without, patients with AI-flagged GOC documentation had higher acuity rates (ICU stay ≥5 days: 30.2% vs 8.2%; in-hospital mortality: 14.9% vs 0.6%; hospice discharge: 10.5% vs 0.4%). Code status changes were also significantly more frequent (29.3% vs 9.9%; p<0.05).

CONCLUSION

Our initiative met the original objectives by providing a financially sustainable EHR optimization that converts unstructured, high-value GOC narratives into real-time discoverable and trackable documentation. Detecting and displaying these conversations is crucial, increasing clinical situational awareness while reducing documentation burden, leading to enhanced goal-concordant care.