Journal of the American Medical Informatics Association : JAMIA · 2026
Benchmarking information extraction of physical activity from electronic health record with large language models: an natural language processing pipeline and comparative evaluation.
СтатьяНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Yang Han, Niu Zhongran, Li Mingchen, Zhou Huixue, Xiao Yongkang, Zhou Sicheng, Zhan Zaifu, Liu Ying, Liu Shiqin, Tignanelli Christopher J, Melton Genevieve B, Zhang Rui
Аннотация
OBJECTIVES: We aimed to develop a data model and a natural language processing (NLP) pipeline for representing physical activity (PA) in Electronic Health Records (EHRs), and to evaluate transformer- and Large Language Model (LLM)-based classifiers for sentence-level PA attribute classification.
MATERIALS AND METHODS: We analyzed PA documentation across three patient cohorts (cancer, COVID, and Alzheimer's disease) using structured and unstructured EHR data. A conceptual schema was developed to represent PA and its linguistic attributes. Five BERT models and three modern LLMs (Llama3-8B, MedAlpaca-13B, and PMC-Llama-13B) were evaluated for classifying PA attributes (binary status, negation, exclusion, and an eleven-class Category) on pre-extracted PA-related sentences.
RESULTS: Clinical notes were a richer source of PA information than structured ICD or SDoH data. On binary tasks, the best BERT model reached F1 0.619 (Exclusion); with Supervised Fine-Tuning (SFT), Llama3-8B reached F1 0.689 (Exclusion). On the 11-class Category task, performance was modest (best macro-F1 0.262, ROC-AUC 0.803, by Llama3-8B).
DISCUSSION: In-Context Learning (ICL) was highly variable: while Llama3-8B-ICL achieved the best ROC-AUC on Category, the domain-specific MedAlpaca-13B and PMC-LLaMA-13B essentially failed. These results, together with sparsely represented PA elements (Amount, Frequency, Assessment) and the absence of a downstream evaluation, position this work as an initial proof of feasibility, with supervised domain adaptation still required for reliable clinical PA extraction.
CONCLUSION: We contribute a PA data model, annotation schema, and a working NLP pipeline with a BERT/LLM benchmark for sentence-level PA attribute classification. The pipeline supports future end-to-end PA extraction and downstream applications such as phenotyping, risk prediction, and cohort identification.
MATERIALS AND METHODS: We analyzed PA documentation across three patient cohorts (cancer, COVID, and Alzheimer's disease) using structured and unstructured EHR data. A conceptual schema was developed to represent PA and its linguistic attributes. Five BERT models and three modern LLMs (Llama3-8B, MedAlpaca-13B, and PMC-Llama-13B) were evaluated for classifying PA attributes (binary status, negation, exclusion, and an eleven-class Category) on pre-extracted PA-related sentences.
RESULTS: Clinical notes were a richer source of PA information than structured ICD or SDoH data. On binary tasks, the best BERT model reached F1 0.619 (Exclusion); with Supervised Fine-Tuning (SFT), Llama3-8B reached F1 0.689 (Exclusion). On the 11-class Category task, performance was modest (best macro-F1 0.262, ROC-AUC 0.803, by Llama3-8B).
DISCUSSION: In-Context Learning (ICL) was highly variable: while Llama3-8B-ICL achieved the best ROC-AUC on Category, the domain-specific MedAlpaca-13B and PMC-LLaMA-13B essentially failed. These results, together with sparsely represented PA elements (Amount, Frequency, Assessment) and the absence of a downstream evaluation, position this work as an initial proof of feasibility, with supervised domain adaptation still required for reliable clinical PA extraction.
CONCLUSION: We contribute a PA data model, annotation schema, and a working NLP pipeline with a BERT/LLM benchmark for sentence-level PA attribute classification. The pipeline supports future end-to-end PA extraction and downstream applications such as phenotyping, risk prediction, and cohort identification.