medRxiv : the preprint server for health sciences · 2026
Multidimensional Assurance of an Electronic Health Record Embedded Generative Artificial Intelligence Summarization Tool.
СтатьяПрепринтНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Bodanki Pavan, Capone Christopher, Hazare Nimay S, Agaron Shamay, Vijayaraghavan Mahima, Japa Sohan, Zaretsky Irina, Epstein Jeffrey, Sawant Ashwin, Shaikh Ahmed, Leibner Evan, Sharma Aniket, Gangadharan Sandeep, Ghani Timsal, Tai Hua-Hsin, Ngai Daniel, El-Haj Sean, Parchure Prathamesh, Itwaru Annelia, Kaplan Ben, Vakil Aditi, Tamler Ronald, Klein Avniel, Freeman Robert, Kovatch Patricia, Darrow Bruce, McGreevy Jolion, Stump Lisa, Nadkarni Girish N, Timsina Prem, Sakhuja Ankit
Аннотация
OBJECTIVE: To evaluate Epic IP Insights, an electronic health record integrated generative AI summarizer used across a seven-hospital health system, using a multidimensional assurance framework.
MATERIALS AND METHODS: We developed and validated an agentic hallucination detector that decomposed summaries into atomic content units (ACUs) and verified each against source documentation. We then applied it to remaining summaries to estimate the hallucination rate. We also measured source note utilization, textual and semantic similarity of regenerated summaries, and clinician perceptions through structured evaluations.
RESULTS: The EPIC IP Insights generated 706 summaries across 445 encounters and used a mean of 5.8% of available notes. The detector was developed on 30 summaries (2,012 ACUs) and validated on 5 held out summaries (295 ACUs). The detector agreed with physician adjudication on 96.95% ACUs in the validation set. Across 671 remaining summaries containing 40,452 ACUs, the hallucination rate was 11.79% (95% CI, 11.47% to 12.10%). Among 385 regenerated pairs of summaries, 32.7% were textually identical and 37.9% were semantically identical. Clinicians found the tool easy to use (97.8%), while responses were more mixed regarding reliance on the output with little verification (45.9%), expected efficiency gains (45.3%), and frequent use (46.1%).
DISCUSSION: Clinicians found the tool easy to use but believed that outputs required verification. Systematic evaluation identified the frequency of hallucinations and limited use of available notes which provide directions for future improvement.
CONCLUSION: Multidimensional assurance frameworks are needed to evaluate the safety, reliability, and consistency of generative AI tools.
MATERIALS AND METHODS: We developed and validated an agentic hallucination detector that decomposed summaries into atomic content units (ACUs) and verified each against source documentation. We then applied it to remaining summaries to estimate the hallucination rate. We also measured source note utilization, textual and semantic similarity of regenerated summaries, and clinician perceptions through structured evaluations.
RESULTS: The EPIC IP Insights generated 706 summaries across 445 encounters and used a mean of 5.8% of available notes. The detector was developed on 30 summaries (2,012 ACUs) and validated on 5 held out summaries (295 ACUs). The detector agreed with physician adjudication on 96.95% ACUs in the validation set. Across 671 remaining summaries containing 40,452 ACUs, the hallucination rate was 11.79% (95% CI, 11.47% to 12.10%). Among 385 regenerated pairs of summaries, 32.7% were textually identical and 37.9% were semantically identical. Clinicians found the tool easy to use (97.8%), while responses were more mixed regarding reliance on the output with little verification (45.9%), expected efficiency gains (45.3%), and frequent use (46.1%).
DISCUSSION: Clinicians found the tool easy to use but believed that outputs required verification. Systematic evaluation identified the frequency of hallucinations and limited use of available notes which provide directions for future improvement.
CONCLUSION: Multidimensional assurance frameworks are needed to evaluate the safety, reliability, and consistency of generative AI tools.