Journal of the American Medical Informatics Association : JAMIA · 2026
Comparative evaluation of ambient digital scribe systems in clinical documentation.
СтатьяНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Weiss Everett, Zsenits Balazs, Nasar Farhad, Phillips Matt, Mudgundi Venugopal, Salhab Tamer, Muth Andrew, Zhu Kaiwen, Kolosko Joanna, Hansen Sean, Bui Quang Neo, Yaseen Ahmad, Balboul Mostafa, Lee Jee, Nunez Isaac Orlando Andres
Аннотация
OBJECTIVES: To demonstrate a replicable in-house framework for evaluating ambient digital scribe (ADS) systems and, applying it, to assess 3 commercial systems and examine alignment between clinician Likert ratings and a manual transcript-level error audit.
MATERIALS AND METHODS: Two standardized clinical encounters-a brief problem-focused visit and a comprehensive annual visit-were simulated and audio-recorded. Three ADS vendors generated notes from identical recordings. A multidisciplinary panel of 56 physicians rated the notes on 7-point Likert scales across 4 domains: relevance, accuracy, quality, and provider confidence. In parallel, physician reviewers cross-referenced each note against the source recording, enumerating hallucinations, omissions, and inaccuracies using a pre-specified rubric.
RESULTS: In this exploratory sample, Vendor A received the highest mean Likert ratings across all 4 domains (Friedman χ²(2) = 15.529, P < .001 short visit; χ²(2) = 10.753, P = .005 long visit). After Bonferroni correction, only Vendor A vs Vendor B on the short visit remained significant (P = .0098). The audit identified fewest errors in Vendor B's notes (n = 3), followed by Vendor A (n = 8) and Vendor C (n = 9). Physicians flagged hallucinations more often than the audit confirmed, a pattern we tentatively label "pseudo-hallucinations."
DISCUSSION: In this sample, clinician preference did not track audit-identified error counts, raising the hypothesis that note-style features influence ratings alongside documentation correctness; larger samples are required.
CONCLUSION: Pairing clinician Likert assessments with transcript-level error audits may be a useful adjunct in evaluating ADS and other generative AI clinical tools; confirmation in larger studies is needed before this framework drives procurement decisions.
MATERIALS AND METHODS: Two standardized clinical encounters-a brief problem-focused visit and a comprehensive annual visit-were simulated and audio-recorded. Three ADS vendors generated notes from identical recordings. A multidisciplinary panel of 56 physicians rated the notes on 7-point Likert scales across 4 domains: relevance, accuracy, quality, and provider confidence. In parallel, physician reviewers cross-referenced each note against the source recording, enumerating hallucinations, omissions, and inaccuracies using a pre-specified rubric.
RESULTS: In this exploratory sample, Vendor A received the highest mean Likert ratings across all 4 domains (Friedman χ²(2) = 15.529, P < .001 short visit; χ²(2) = 10.753, P = .005 long visit). After Bonferroni correction, only Vendor A vs Vendor B on the short visit remained significant (P = .0098). The audit identified fewest errors in Vendor B's notes (n = 3), followed by Vendor A (n = 8) and Vendor C (n = 9). Physicians flagged hallucinations more often than the audit confirmed, a pattern we tentatively label "pseudo-hallucinations."
DISCUSSION: In this sample, clinician preference did not track audit-identified error counts, raising the hypothesis that note-style features influence ratings alongside documentation correctness; larger samples are required.
CONCLUSION: Pairing clinician Likert assessments with transcript-level error audits may be a useful adjunct in evaluating ADS and other generative AI clinical tools; confirmation in larger studies is needed before this framework drives procurement decisions.