External Validation of a Machine Learning Based Delirium Prediction Software in Clinical Routine.

BACKGROUND

Various machine learning (ML) models have been developed for the prediction of clinical outcomes, but there is missing evidence on their performance in clinical routine and external validation.

OBJECTIVES

Our aim was to deploy and prospectively evaluate an already developed delirium prediction software in clinical routine of an external hospital.

METHODS

We compared updated ML models of the software and models re-trained with the external hospital's data. The best models were deployed in clinical routine for one month, and risk predictions for all admitted patients were compared to the risk ratings of a senior physician. After using the software, clinicians completed a questionnaire assessing technology acceptance.

RESULTS

Re-trained models achieved a high discriminative performance (AUROC > 0.92). Compared to clinical risk ratings, the software achieved a sensitivity of 100.0% and a specificity of 90.6%. Usefulness, ease of use and output quality were rated positively by the users.

CONCLUSION

A ML based delirium prediction software achieved a high discriminative performance and high technology acceptance at an external hospital using re-trained ML models.