JAMIA open · 2026
Development of a benchmarking dataset for symptom detection using large language models.
СтатьяНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Davis Joshua, Durieux Brigitte N, Van Dongen Chloe, Sciacca Kate, Gutierrez Juan Manuel, Lindvall Charlotta
Аннотация
OBJECTIVES: To develop a pipeline for evaluating large language models (LLMs) on the task of capturing symptoms from clinical encounters.
MATERIALS AND METHODS: We created a gold standard dataset of symptom annotations from simulated doctor-patient encounter excerpts (264 encounters; 16 symptoms; double-coded and adjudicated). Nine different LLMs from 4 vendors (OpenAI, Meta, DeepSeek, Moonshot AI) were used as examples to test our evaluation pipeline; outputs were assessed for correct structure and symptom information.
RESULTS: Of 3085 excerpts, 2087 (68%) contained symptoms. Pain, cough, and shortness of breath were most common; LLMs achieved F1 scores ranging 0.66-0.88 for these symptoms with minimal prompt engineering. Of tested models, GPT-4.1 demonstrated the best overall performance.
DISCUSSION: Our evaluation pipeline and benchmarking dataset are publicly available and applicable to various LLMs, including open-source models.
CONCLUSION: This work supports the development and optimization of models that seek to improve patient symptom understanding.
MATERIALS AND METHODS: We created a gold standard dataset of symptom annotations from simulated doctor-patient encounter excerpts (264 encounters; 16 symptoms; double-coded and adjudicated). Nine different LLMs from 4 vendors (OpenAI, Meta, DeepSeek, Moonshot AI) were used as examples to test our evaluation pipeline; outputs were assessed for correct structure and symptom information.
RESULTS: Of 3085 excerpts, 2087 (68%) contained symptoms. Pain, cough, and shortness of breath were most common; LLMs achieved F1 scores ranging 0.66-0.88 for these symptoms with minimal prompt engineering. Of tested models, GPT-4.1 demonstrated the best overall performance.
DISCUSSION: Our evaluation pipeline and benchmarking dataset are publicly available and applicable to various LLMs, including open-source models.
CONCLUSION: This work supports the development and optimization of models that seek to improve patient symptom understanding.