DOI 10.64898/2026.08.31.26361660
Multidimensional Assurance of an Electronic Health Record Embedded Generative Artificial Intelligence Summarization Tool.
Аннотация
OBJECTIVE
To evaluate Epic IP Insights, an electronic health record integrated generative AI summarizer used across a seven-hospital health system, using a multidimensional assurance framework.
MATERIALS AND
METHODS
We developed and validated an agentic hallucination detector that decomposed summaries into atomic content units (ACUs) and verified each against source documentation. We then applied it to remaining summaries to estimate the hallucination rate. We also measured source note utilization, textual and semantic similarity of regenerated summaries, and clinician perceptions through structured evaluations.
RESULTS
The EPIC IP Insights generated 706 summaries across 445 encounters and used a mean of 5.8% of available notes. The detector was developed on 30 summaries (2,012 ACUs) and validated on 5 held out summaries (295 ACUs). The detector agreed with physician adjudication on 96.95% ACUs in the validation set. Across 671 remaining summaries containing 40,452 ACUs, the hallucination rate was 11.79% (95% CI, 11.47% to 12.10%). Among 385 regenerated pairs of summaries, 32.7% were textually identical and 37.9% were semantically identical. Clinicians found the tool easy to use (97.8%), while responses were more mixed regarding reliance on the output with little verification (45.9%), expected efficiency gains (45.3%), and frequent use (46.1%).
DISCUSSION: Clinicians found the tool easy to use but believed that outputs required verification. Systematic evaluation identified the frequency of hallucinations and limited use of available notes which provide directions for future improvement.
CONCLUSION
Multidimensional assurance frameworks are needed to evaluate the safety, reliability, and consistency of generative AI tools.
Подготовка и перевод публикации
Обработка статьи · этапы, ошибки и управление
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Тип публикации указан по данным источника. Уровень доказательности автоматически не присваивается.