Eye (London, England) · 2026
Large language model-based scribing tools for ophthalmology: performance and safety evaluation using simulated clinical encounters.
СтатьяСравнительное исследованиеНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Zou Minjie, Srinivasan Sahana, Chin Victoria, Lo Thaddaeus Wai Soon, Loke Wan Ting, Yang Gabriel Dawei, Lim Zhi Wei, Li Kelvin Zhenghao, Wong Wendy Meihua, Lam Janice, Chen David Ziyou, Tham Yih Chung
Аннотация
BACKGROUND: To compare the performance and safety of six LLM-based scribing tools in generating ophthalmological clinical notes from simulated physician-patient encounters.
METHODS: This cross-sectional comparative study involved seven ophthalmic encounters, transcribed by six scribing tools (Heidi AI, Whisper GPT-4o, Whisper Gemini 1.5 Pro, i-scribe, Lyrebird, Twofold). Scribing tools were prompted to generate notes using Subjective, Objective, Assessment, and Plan (SOAP) format. Consultant-level ophthalmologists evaluated note quality using the Physician Documentation Quality Instrument (PDQI-9), which evaluates eight dimensions: "accurate", "thorough", "useful", "organized", "comprehensive", "succinct", "synthesized", "internally consistent" on a 5-point scale (with a maximum sum of 40). Text-generation metrics (ROUGE-L, BERTScore, BARTScore, AlignScore) were also used. Safety analysis was conducted to identify documentation errors categorised as omissions, incorrect information, and extraneous content.
RESULTS: Heidi AI achieved the highest overall documentation quality (PDQI-9: 38.524, 95%CI: 37.815-39.233), followed by Whisper GPT-4o (38.048, 95%CI: 36.747-39.349). It scored highest in accuracy, usefulness, organisation, comprehensiveness, and succinctness, and led in ROUGE-L (0.326, 95%CI: 0.276-0.374), BERTScore (0.727, 95%CI: 0.707-0.747), and AlignScore (0.704, 95%CI: 0.650-0.758). All tools, however, exhibited errors. The most common were omissions of clinically relevant details, including disease severity, coexisting ocular conditions, systemic comorbidities, and management specifics. Some tools also generated incorrect or extraneous content, such as hallucinated patient requests or incorrect patient names.
CONCLUSIONS: Among the evaluated LLM based scribing tools, Heidi AI demonstrated the highest performance and shows promise for supporting ophthalmology documentation. However, persistent errors, particularly omissions of critical details, were observed across all tools and could compromise patient care. Human oversight and rigorous verification processes remain essential before clinical integration.
METHODS: This cross-sectional comparative study involved seven ophthalmic encounters, transcribed by six scribing tools (Heidi AI, Whisper GPT-4o, Whisper Gemini 1.5 Pro, i-scribe, Lyrebird, Twofold). Scribing tools were prompted to generate notes using Subjective, Objective, Assessment, and Plan (SOAP) format. Consultant-level ophthalmologists evaluated note quality using the Physician Documentation Quality Instrument (PDQI-9), which evaluates eight dimensions: "accurate", "thorough", "useful", "organized", "comprehensive", "succinct", "synthesized", "internally consistent" on a 5-point scale (with a maximum sum of 40). Text-generation metrics (ROUGE-L, BERTScore, BARTScore, AlignScore) were also used. Safety analysis was conducted to identify documentation errors categorised as omissions, incorrect information, and extraneous content.
RESULTS: Heidi AI achieved the highest overall documentation quality (PDQI-9: 38.524, 95%CI: 37.815-39.233), followed by Whisper GPT-4o (38.048, 95%CI: 36.747-39.349). It scored highest in accuracy, usefulness, organisation, comprehensiveness, and succinctness, and led in ROUGE-L (0.326, 95%CI: 0.276-0.374), BERTScore (0.727, 95%CI: 0.707-0.747), and AlignScore (0.704, 95%CI: 0.650-0.758). All tools, however, exhibited errors. The most common were omissions of clinically relevant details, including disease severity, coexisting ocular conditions, systemic comorbidities, and management specifics. Some tools also generated incorrect or extraneous content, such as hallucinated patient requests or incorrect patient names.
CONCLUSIONS: Among the evaluated LLM based scribing tools, Heidi AI demonstrated the highest performance and shows promise for supporting ophthalmology documentation. However, persistent errors, particularly omissions of critical details, were observed across all tools and could compromise patient care. Human oversight and rigorous verification processes remain essential before clinical integration.