Clinical heterogeneity in performance of a commercial prediction model for clinical deterioration.

OBJECTIVES

Although the Epic Deterioration Index (EDI) model is widely used in US hospitals, heterogeneity in its performance among patient subgroups remains unclear. Our objective was to assess EDI performance when applied across clinical service lines and hospital locations to inform clinical utility.

METHODS

This retrospective cross-sectional study evaluated EDI performance among adult inpatients at a Northern California academic hospital from 2021 to 2024. The EDI score, calculated every 15 min, was assessed for its ability to predict deterioration events defined as intensive care unit transfer, rapid response team activation, comfort care initiation or death. Model performance was quantified using the area under the receiver operating characteristic curve (AUROC), calibration curves and confusion matrix statistics using individual EDI scores and an event horizon of 12 hours. Sensitivity analysis considered event horizons of 2, 4, 8 and 24 hours.

RESULTS

Among 77 478 encounters, 10 579 events occurred, with an overall AUROC of 0.763 (95% CI 0.759 to 0.768). Performance varied by clinical service, highest in haematology/oncology/bone marrow transplant (AUROC 0.815) and lowest in cardiology (0.670) and neurosciences (0.677). Similar variability was observed across hospital locations.

DISCUSSION: Considerable variability in predictive performance was found across clinical subpopulations in a hospital setting, highlighting the limitations of the common practice of hospital-wide deployment of predictive models.

CONCLUSION

Understanding the differential performance of predictive models among clinically relevant patient subgroups is paramount to the safe, effective and precise deployment of models in the inpatient setting.