Journal of medical systems · 2026
Calibration of Self-Reported Confidence and Accuracy of Large Language Models in Medical Question Answering.
СтатьяНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Boie Sebastian Daniel, Reis Florian, Frey Nicolas, Grünewald Elias, Balzer Felix
Аннотация
OBJECTIVE: To evaluate the alignment between self-reported confidence of large language models (LLM) and their accuracy in answering medical multiple-choice questions.
MATERIALS AND METHODS: We prompted six LLMs (GPT-5, GPT-5-mini, GPT-5-nano, GPT-4o, Claude Sonnet 4.5, and Gemini 2.5) to answer MedMCQA items and report confidence scores. Based on 12,000 LLM responses, we calculated Expected Calibration Errors (ECE) by averaging absolute differences between observed accuracy and predicted confidence.
RESULTS: Mean ECE differed by model (Claude Sonnet 4.5 best: 0.06; Gpt-4o worst: 0.127) and varied across specialties ("Skin" best: 0.041; "Social & Preventive Medicine" worst: 0.141). Accuracy of examined LLMs showed analogous variation between specialties.
DISCUSSION: Our results demonstrate that high accuracy does not guarantee reliable uncertainty estimation. We identified substantial heterogeneity across medical specialties, where pooled metrics masked a threefold ECE increase between best- and worst-performing domains.
CONCLUSION: We recommend incorporating calibration reporting into LLM evaluations, as larger models exhibit improved "self-knowledge", but uneven overconfidence persists.
MATERIALS AND METHODS: We prompted six LLMs (GPT-5, GPT-5-mini, GPT-5-nano, GPT-4o, Claude Sonnet 4.5, and Gemini 2.5) to answer MedMCQA items and report confidence scores. Based on 12,000 LLM responses, we calculated Expected Calibration Errors (ECE) by averaging absolute differences between observed accuracy and predicted confidence.
RESULTS: Mean ECE differed by model (Claude Sonnet 4.5 best: 0.06; Gpt-4o worst: 0.127) and varied across specialties ("Skin" best: 0.041; "Social & Preventive Medicine" worst: 0.141). Accuracy of examined LLMs showed analogous variation between specialties.
DISCUSSION: Our results demonstrate that high accuracy does not guarantee reliable uncertainty estimation. We identified substantial heterogeneity across medical specialties, where pooled metrics masked a threefold ECE increase between best- and worst-performing domains.
CONCLUSION: We recommend incorporating calibration reporting into LLM evaluations, as larger models exhibit improved "self-knowledge", but uneven overconfidence persists.