Developing the CAM-BERT: Enhancing delirium screening in hospitalized older adults using natural language processing.

BACKGROUND

Delirium is a common condition affecting hospitalized older adults, often leading to adverse outcomes. Nevertheless, delirium frequently goes unrecognized due to various clinical and systemic challenges. We aimed to develop and evaluate a deep-learning natural language processing (NLP) model trained on Brazilian Portuguese clinical notes, aiming to improve the identification of delirium symptoms in electronic health records (EHR) and to facilitate the detection of delirium.

METHODS

We extracted free-text clinical notes from 500 hospitalizations of older adults at a tertiary care hospital in São Paulo, Brazil, for annotation and analysis. Delirium symptoms were identified and labeled by expert clinicians using a structured protocol. The deep learning model BERTimbau was employed alongside a classical Random Forest approach for comparison, with performance metrics derived from F1 scores. We also developed the CAM-BERT framework, an algorithmic approach to categorize potential delirium cases by aligning symptoms classified by the model with CAM criteria.

RESULTS

The BERTimbau model showcased superior performance, with an F1-macro score of 77 % compared to the baseline model's 39 %. It achieved F1 scores of around 90 % for identifying confusion and disorganized thinking. For the CAM-BERT framework, which mapped detected symptoms to CAM criteria, the overall F1-macro score was 83 %. The agreement with expert chart review yielded a Cohen's kappa coefficient of 0.72.

CONCLUSIONS

The study highlights the potential of artificial intelligence, particularly NLP, in supporting the recognition of delirium in non-English-speaking clinical settings. Further research is needed to validate the model's applicability across diverse healthcare environments.