European journal of radiology · 2025
Evaluation of DeepSeek-R1 and contemporary large language models on the radiology board examination: A milestone achieved as open-source model matches performance with closed-source model.
СтатьяНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Nakaura Takeshi, Kobayashi Naoki, Masuda Takanori, Nagayama Yasunori, Uetani Hiroyuki, Kidoh Masafumi, Oda Seitaro, Funama Yoshinori, Hirai Toshinori
Аннотация
OBJECTIVES: Recent advances in large language models (LLMs), especially reasoning LLMs have demonstrated impressive reasoning capabilities in specialized domains. The purpose of this study is to evaluate the performance of new open-source reasoning LLM, DeepSeek-R1 and other contemporary LLMs on radiology board examination questions, comparing their accuracy to human radiologists.
MATERIALS AND METHODS: We assessed 10 LLMs, including both closed-source models (GPT-4o, o1, o3-mini, Claude 3.5 Sonnet, Gemini Flash 2.0) and open-source models (DeepSeek-R1, its distilled versions, and Llama 3.3 70B), on 105 non-image multiple-choice questions from the 2024 official board examination of the Japan Radiological Society. We evaluated accuracies of LLMs and compared those to accuracies of examinees (3rd-year radiology residents).
RESULTS: DeepSeek-R1 and OpenAI o1 each answered 92/105 items correctly (87.6 %, 95 % CI: 80.1-92.5); McNemar's test showed no significant head-to-head difference (p = 1.00), outperforming human radiologists' mean accuracy of 67.6 ± 9.3 % (z-score = 2.15, equivalent to 98.4th percentile). DeepSeek-R1 demonstrated superior cost-efficiency with API costs approximately 1/27th of o1. DeepSeek-R1's distilled 32B model (61.9 % accuracy) outperformed Llama 3.3 70B (57.1 %), despite having fewer parameters.
CONCLUSION: Open-source DeepSeek-R1 matches the performance of top closed-source models while offering superior cost-efficiency in radiology knowledge assessment.
MATERIALS AND METHODS: We assessed 10 LLMs, including both closed-source models (GPT-4o, o1, o3-mini, Claude 3.5 Sonnet, Gemini Flash 2.0) and open-source models (DeepSeek-R1, its distilled versions, and Llama 3.3 70B), on 105 non-image multiple-choice questions from the 2024 official board examination of the Japan Radiological Society. We evaluated accuracies of LLMs and compared those to accuracies of examinees (3rd-year radiology residents).
RESULTS: DeepSeek-R1 and OpenAI o1 each answered 92/105 items correctly (87.6 %, 95 % CI: 80.1-92.5); McNemar's test showed no significant head-to-head difference (p = 1.00), outperforming human radiologists' mean accuracy of 67.6 ± 9.3 % (z-score = 2.15, equivalent to 98.4th percentile). DeepSeek-R1 demonstrated superior cost-efficiency with API costs approximately 1/27th of o1. DeepSeek-R1's distilled 32B model (61.9 % accuracy) outperformed Llama 3.3 70B (57.1 %), despite having fewer parameters.
CONCLUSION: Open-source DeepSeek-R1 matches the performance of top closed-source models while offering superior cost-efficiency in radiology knowledge assessment.