Transporting a pediatric surgical site infection prediction model to the electronic health record: external validation, harmonization, and recalibration.

OBJECTIVES

To externally validate a pediatric surgical site infection (SSI) prediction model, assess institutional and temporal transportability, and evaluate recalibration strategies to support adaptation of registry-derived predictions for clinical decision support (CDS).

MATERIALS AND

METHODS

An elastic-net logistic regression model developed using a national pediatric surgical registry (2012-2022) was validated in an institutional electronic health record (EHR)-derived cohort (10 450 procedures) and in a newly released, temporally distinct cohort from the same registry (301 999 procedures). We assessed discrimination, calibration, and overall performance, used a closed testing procedure to guide recalibration, and evaluated clinical utility using decision curve analysis.

RESULTS

Performance was preserved in the registry cohort, with c-statistic 0.75, Brier score 0.03, calibration slope 1.00, and observed/expected (O/E) ratio 1.01. Direct application to the institutional EHR cohort produced inflated predicted risk and poor performance, with c-statistic 0.60, Brier score 0.07, calibration slope 0.15, and O/E ratio 0.25. After omitting open-ended diagnosis-code indicators, performance improved, with c-statistic 0.70, Brier score 0.03, calibration slope 0.81, and O/E ratio 1.35. Logistic recalibration restored calibration slope and O/E ratio to 1.00. Decision curve analysis showed positive net benefit across prespecified thresholds.

DISCUSSION: These findings highlight a key challenge for clinical translation: models that perform well under internal or registry-based validation may not be directly transportable to real-world EHR environments. Predictor misalignment-differences in how model predictors were captured and measured across registry and EHR data-emerged as a key barrier to direct institutional deployment, underscoring the importance of implementation-aware external validation before clinical use.

CONCLUSION

Addressing diagnosis-code predictor misalignment and applying parsimonious recalibration supported institutional adaptation of a pediatric SSI prediction model, providing a pragmatic pathway for translating registry-derived models into real-world clinical practice. This work sets the foundation for prospective evaluation of EHR-embedded CDS to enable individualized SSI risk estimation at the point of care.