Digestive and liver disease : official journal of the Italian Society of Gastroenterology and the Italian Association for the Study of the Liver · 2025
Performance of gastroenterologists and multimodal LLMs in endoscopic EREFS scoring of Eosinophilic Esophagitis.
СтатьяНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Levartovsky Asaf, Dar Lior, Fay Shmuel, Ukashi Offir, Shachar Eyal, Engel Tal, Ben-Horin Shomron, Savarino Edoardo V, Ungar Bella
Аннотация
BACKGROUND: The comprehensive evaluation of multimodal large language models (LLMs) in gastroenterological image-analysis remains limited. We aimed to assess the accuracy of the Eosinophilic Esophagitis Endoscopic Reference Score (EREFS) scoring system for Eosinophilic Esophagitis (EoE) across gastroenterology (GI) clinicians with varying levels of experience.
METHODS: Fifty real-world endoscopic images of EoE patients were graded based on the original EREFS score by a gold standard anchor-rater. EREFS gradings of GI specialists, fellows and three multimodal LLMs (ChatGPT-4o, Claude Sonnet 3.5, Perplexity Sonar) were compared with the anchor-rater's scoring. LLMs were provided with both single-shot and few-shot prompting strategies to optimize performance.
RESULTS: Overall assessment accuracy was significantly higher among GI fellows (72.4 %) compared to specialists (65.3 %, p = 0.004) and LLMs (58.9 %, p < 0.001). In the detection of edema, LLMs outperformed specialists (83.3 % vs 49.3 %, p < 0.001). However, LLMs showed significantly poor performance in rings assessment (30 %) compared to specialists (58 %) and fellows (58.7 %, both p < 0.001). After implementation of few-shot prompting, the overall performance of LLMs was comparable to GI specialists (62.7 % vs 65.3 %, p = 0.3).
CONCLUSIONS: This study uncovers variability in EREFS scoring across human raters with different expertise and multimodal LLMs, and demonstrates that few-shot prompting can optimize LLM accuracy.
METHODS: Fifty real-world endoscopic images of EoE patients were graded based on the original EREFS score by a gold standard anchor-rater. EREFS gradings of GI specialists, fellows and three multimodal LLMs (ChatGPT-4o, Claude Sonnet 3.5, Perplexity Sonar) were compared with the anchor-rater's scoring. LLMs were provided with both single-shot and few-shot prompting strategies to optimize performance.
RESULTS: Overall assessment accuracy was significantly higher among GI fellows (72.4 %) compared to specialists (65.3 %, p = 0.004) and LLMs (58.9 %, p < 0.001). In the detection of edema, LLMs outperformed specialists (83.3 % vs 49.3 %, p < 0.001). However, LLMs showed significantly poor performance in rings assessment (30 %) compared to specialists (58 %) and fellows (58.7 %, both p < 0.001). After implementation of few-shot prompting, the overall performance of LLMs was comparable to GI specialists (62.7 % vs 65.3 %, p = 0.3).
CONCLUSIONS: This study uncovers variability in EREFS scoring across human raters with different expertise and multimodal LLMs, and demonstrates that few-shot prompting can optimize LLM accuracy.