JAMIA open · 2026
Development and evaluation of machine learning models for the detection of emergency department patients with opioid misuse from clinical notes.
СтатьяНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Shahid Usman, Parde Natalie, Smith Dale L, Dickinson Grayson, Bianco Joseph, Thorpe Dillon, Hota Madhav, Afshar Majid, Karnik Niranjan S, Chhabra Neeraj
Аннотация
OBJECTIVES: The accurate identification of Emergency Department (ED) encounters involving opioid misuse is critical for health services, research, and surveillance. We sought to develop natural language processing (NLP)-based models for the detection of ED encounters involving opioid misuse.
METHODS: A sample of ED encounters enriched for opioid misuse was manually annotated and clinical notes extracted. We evaluated classic machine learning (ML) methods, fine-tuning of publicly available pretrained language models, and a previously developed convolutional neural network opioid classifier for use on hospitalized patients (SMART-AI). Performance was benchmarked to opioid-related ICD-10-CM codes. Both raw text and text transformed to the Unified Medical Language System were evaluated. Face validity was evaluated by term feature importance.
RESULTS: There were 1123 encounters used for training, validation, and testing. Of the classic ML methods, XGBoost had the highest AU_PRC 0.9358 (95% CI 0.8945, 0.9681), accuracy 0.8874 (0.8402, 0.9349), and F1 score 0.8624 (0.7969, 0.9197) which performed comparably to ICD-10-CM codes [accuracy 0.8687 (0.8155, 0.9167); F1 0.8296 (0.7544, 0.8939)]. Excluding XGBoost, fine-tuned pre-trained language models generally outperformed classic ML methods. The best performing model by point estimate was the fine-tuned SMART-AI based model with domain adaptation [AU_PRC 0.9474 (0.9113, 0.9749); accuracy 0.8816 (0.8284, 0.9290); F1 0.8499 (0.7805, 0.9103)] but confidence intervals overlapped with other models. Explainability analyses showed the most predictive terms were "heroin," "opioids," "alcoholic intoxication, chronic," "cocaine," "opiates," and "suboxone."
CONCLUSIONS: NLP-based models perform comparably to entry of ICD-10-CM diagnosis codes for the detection of ED encounters with opioid misuse. Fine tuning with domain adaptation for pre-trained language models resulted in improved performance.
METHODS: A sample of ED encounters enriched for opioid misuse was manually annotated and clinical notes extracted. We evaluated classic machine learning (ML) methods, fine-tuning of publicly available pretrained language models, and a previously developed convolutional neural network opioid classifier for use on hospitalized patients (SMART-AI). Performance was benchmarked to opioid-related ICD-10-CM codes. Both raw text and text transformed to the Unified Medical Language System were evaluated. Face validity was evaluated by term feature importance.
RESULTS: There were 1123 encounters used for training, validation, and testing. Of the classic ML methods, XGBoost had the highest AU_PRC 0.9358 (95% CI 0.8945, 0.9681), accuracy 0.8874 (0.8402, 0.9349), and F1 score 0.8624 (0.7969, 0.9197) which performed comparably to ICD-10-CM codes [accuracy 0.8687 (0.8155, 0.9167); F1 0.8296 (0.7544, 0.8939)]. Excluding XGBoost, fine-tuned pre-trained language models generally outperformed classic ML methods. The best performing model by point estimate was the fine-tuned SMART-AI based model with domain adaptation [AU_PRC 0.9474 (0.9113, 0.9749); accuracy 0.8816 (0.8284, 0.9290); F1 0.8499 (0.7805, 0.9103)] but confidence intervals overlapped with other models. Explainability analyses showed the most predictive terms were "heroin," "opioids," "alcoholic intoxication, chronic," "cocaine," "opiates," and "suboxone."
CONCLUSIONS: NLP-based models perform comparably to entry of ICD-10-CM diagnosis codes for the detection of ED encounters with opioid misuse. Fine tuning with domain adaptation for pre-trained language models resulted in improved performance.