Journal of the American Medical Informatics Association : JAMIA · 2026
A language model framework for sequence modeling of audit logs to characterize clinician-electronic health record interactions.
СтатьяНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Kim Seunghwan, Lou Sunny S, Cobb Adam, Jha Susmit, Kannampallil Thomas
Аннотация
OBJECTIVE: Electronic health record (EHR) audit logs capture clinician-EHR interaction patterns, but most audit log research relies on aggregated measures (eg, total time). We investigated how audit logs could be modeled using large language model (LLM) architectures to learn underlying workflow sequences during clinician-EHR interactions.
MATERIALS AND METHODS: Using >295 million EHR-based audit log actions from inpatient settings spanning 2019 to 2024, we fine-tuned Llama-3-8B under 2 encodings: (1) symbolic (field-based) tokens and (2) semantic (word-based) natural language-based audit log action descriptions. A first-order Markov model, using only the immediate prior action, served as a baseline minimal context comparator. Model representation was assessed using next-action prediction accuracy in an early-period test set and 2 temporally distinct out-of-sample (OOS) periods.
RESULTS: In the early-period test set, the word-based LLM achieved the highest accuracy (0.7473; 95% CI, 0.7468-0.7478) compared to the field-based LLM (0.3464; 95% CI, 0.3456-0.3472) and Markov baseline (0.4230; 95% CI, 0.4223-0.4236). The word-based approach retained a substantial performance advantage over the Markov baseline across the OOS periods, although accuracy declined over time (word-based LLM vs Markov accuracy, OOS-1: 0.6555 vs 0.3139, OOS-2: 0.6286 vs 0.2720).
DISCUSSION: Word-based LLM relying on audit log action descriptions yielded the highest next-action prediction accuracy and demonstrated relative robustness to temporal drift, suggesting that longer sequential context and word-based action descriptions may improve audit log-based sequence modeling.
CONCLUSIONS: These findings support the development of natural language-based LLM models for audit log research for task identification, workflow characterization, and safety-focused analyses of clinician-EHR interaction patterns.
MATERIALS AND METHODS: Using >295 million EHR-based audit log actions from inpatient settings spanning 2019 to 2024, we fine-tuned Llama-3-8B under 2 encodings: (1) symbolic (field-based) tokens and (2) semantic (word-based) natural language-based audit log action descriptions. A first-order Markov model, using only the immediate prior action, served as a baseline minimal context comparator. Model representation was assessed using next-action prediction accuracy in an early-period test set and 2 temporally distinct out-of-sample (OOS) periods.
RESULTS: In the early-period test set, the word-based LLM achieved the highest accuracy (0.7473; 95% CI, 0.7468-0.7478) compared to the field-based LLM (0.3464; 95% CI, 0.3456-0.3472) and Markov baseline (0.4230; 95% CI, 0.4223-0.4236). The word-based approach retained a substantial performance advantage over the Markov baseline across the OOS periods, although accuracy declined over time (word-based LLM vs Markov accuracy, OOS-1: 0.6555 vs 0.3139, OOS-2: 0.6286 vs 0.2720).
DISCUSSION: Word-based LLM relying on audit log action descriptions yielded the highest next-action prediction accuracy and demonstrated relative robustness to temporal drift, suggesting that longer sequential context and word-based action descriptions may improve audit log-based sequence modeling.
CONCLUSIONS: These findings support the development of natural language-based LLM models for audit log research for task identification, workflow characterization, and safety-focused analyses of clinician-EHR interaction patterns.