A language model framework for sequence modeling of audit logs to characterize clinician-electronic health record interactions.

OBJECTIVE

Electronic health record (EHR) audit logs capture clinician-EHR interaction patterns, but most audit log research relies on aggregated measures (eg, total time). We investigated how audit logs could be modeled using large language model (LLM) architectures to learn underlying workflow sequences during clinician-EHR interactions.

MATERIALS AND

METHODS

Using >295 million EHR-based audit log actions from inpatient settings spanning 2019 to 2024, we fine-tuned Llama-3-8B under 2 encodings: (1) symbolic (field-based) tokens and (2) semantic (word-based) natural language-based audit log action descriptions. A first-order Markov model, using only the immediate prior action, served as a baseline minimal context comparator. Model representation was assessed using next-action prediction accuracy in an early-period test set and 2 temporally distinct out-of-sample (OOS) periods.

RESULTS

In the early-period test set, the word-based LLM achieved the highest accuracy (0.7473; 95% CI, 0.7468-0.7478) compared to the field-based LLM (0.3464; 95% CI, 0.3456-0.3472) and Markov baseline (0.4230; 95% CI, 0.4223-0.4236). The word-based approach retained a substantial performance advantage over the Markov baseline across the OOS periods, although accuracy declined over time (word-based LLM vs Markov accuracy, OOS-1: 0.6555 vs 0.3139, OOS-2: 0.6286 vs 0.2720).

DISCUSSION: Word-based LLM relying on audit log action descriptions yielded the highest next-action prediction accuracy and demonstrated relative robustness to temporal drift, suggesting that longer sequential context and word-based action descriptions may improve audit log-based sequence modeling.

CONCLUSIONS

These findings support the development of natural language-based LLM models for audit log research for task identification, workflow characterization, and safety-focused analyses of clinician-EHR interaction patterns.