Exposing and addressing the clinical safety gap in ESI-free emergency department triage: Imbalance-aware, interpretable deep learning.

BACKGROUND

AI-based emergency department (ED) triage must balance predictive performance, clinical safety, and interpretability. Prior models suffer from target leakage due to Emergency Severity Index (ESI) scores, or prioritize accuracy while failing to detect high-risk patients.

OBJECTIVE

To expose a pervasive clinical safety gap in ESI-free ED triage, accuracy-optimized models detecting fewer than 1% of high-risk patients despite AUROC near 0.80, and to show that imbalance-aware training is a broadly effective remedy across the evaluated model families.

METHODS

Using MIMIC-IV-ED (N = 407,735 adult visits; ESI excluded from inputs and labels), outcome-based labels were High Risk (expired, transferred, or 30-day mortality), Medium (admitted, survived), and Low (discharged). DA-V2, a dual-attention model trained with focal loss, mixup, and five-seed ensembling, was compared against seven baselines and an identically-trained MLP; a vitals-only variant was externally validated. Attention-SHAP concordance was assessed by Spearman correlation.

RESULTS

DA-V2 achieved AUROC 0.7949 (95% CI 0.7908-0.7991), outperforming all balanced tree baselines (p<0.001). An identically-trained MLP matched it (AUROC 0.7937, p=0.928; HR Recall 74.2% vs. 74.1%), showing that imbalance-aware training, not the architecture, drives detection. Focal loss with seed ensembling (mixup omitted, TabNet API) raised TabNet's HR Recall from under 1% to 44% (comparable to class-balanced TabNet, 45%) but did not reach the 74% attained only by the two dense models. Default models detected fewer than 1% of high-risk patients despite AUROC 0.796-0.800. Externally (NHAMCS-ED 2022, 12,598 adults; an approximate, proxy-label check), the vitals-only model retained HR Recall 0.759 at AUROC 0.695. Attention showed rank concordance with SHAP (Spearman ρ=0.708, p<10-12), evidence of rank agreement rather than causal faithfulness.

CONCLUSIONS

A pervasive clinical safety gap, high AUROC with near-zero high-risk detection, is closed by imbalance-aware training and operating-point selection, not by any specific architecture. DA-V2 provides an interpretable, externally checked instantiation for trustworthy, ESI-free triage.