Evaluation of DeepSeek-R1 and contemporary large language models on the radiology board examination: A milestone achieved as open-source model matches performance with closed-source model.

OBJECTIVES

Recent advances in large language models (LLMs), especially reasoning LLMs have demonstrated impressive reasoning capabilities in specialized domains. The purpose of this study is to evaluate the performance of new open-source reasoning LLM, DeepSeek-R1 and other contemporary LLMs on radiology board examination questions, comparing their accuracy to human radiologists.

MATERIALS AND

METHODS

We assessed 10 LLMs, including both closed-source models (GPT-4o, o1, o3-mini, Claude 3.5 Sonnet, Gemini Flash 2.0) and open-source models (DeepSeek-R1, its distilled versions, and Llama 3.3 70B), on 105 non-image multiple-choice questions from the 2024 official board examination of the Japan Radiological Society. We evaluated accuracies of LLMs and compared those to accuracies of examinees (3rd-year radiology residents).

RESULTS

DeepSeek-R1 and OpenAI o1 each answered 92/105 items correctly (87.6 %, 95 % CI: 80.1-92.5); McNemar's test showed no significant head-to-head difference (p = 1.00), outperforming human radiologists' mean accuracy of 67.6 ± 9.3 % (z-score = 2.15, equivalent to 98.4th percentile). DeepSeek-R1 demonstrated superior cost-efficiency with API costs approximately 1/27th of o1. DeepSeek-R1's distilled 32B model (61.9 % accuracy) outperformed Llama 3.3 70B (57.1 %), despite having fewer parameters.

CONCLUSION

Open-source DeepSeek-R1 matches the performance of top closed-source models while offering superior cost-efficiency in radiology knowledge assessment.