Large Language Model Cost and Performance: A Comprehensive Analysis in the Context of the Japan Radiology Board Examination.

OBJECTIVES

This study aims to evaluate various large language models (LLMs) for their effectiveness in answering Japan Radiology Board Examination (JRBE).

MATERIALS AND

METHODS

A total of 315 examination questions from 2021 to 2023 JRBE were administered to 14 LLMs, comprising 7 open-source and 7 closed-source models. Each model processed the questions in their original Japanese language and after translating them into English by LLMs themselves. Performance metrics, including median scores, interquartile ranges (IQR), P -values, and correlation coefficients, were analyzed using Python.

RESULTS

Closed-source models achieved a higher median correct response rate of 44.28% (IQR: 32.38%-53.02%) compared with open-source models at 29.52% (IQR: 24.76%-36.67%), exhibiting a 50.2% improvement ( P < 0.001). Translating questions to English improved performance, with median scores increasing from 33.02% (IQR: 27.86%-43.65%) to 40.00% (IQR: 27.54%-46.27%), representing a 21.1% increase ( P = 0.005). A positive correlation was observed between the cost per 1M tokens and accuracy (Japanese: r = 0.623, P = 0.017; English: r = 0.613, P = 0.020). No significant correlation was found between model release dates and performance. Only the high-cost closed-source models, GPT-4 and Claude-3-Opus, surpassed the 60% passing threshold by Japanese Medical Specialty Board when using English translation. Among open-source models, LLama-3-70B demonstrated notable performance improvements compared with its predecessors.

CONCLUSIONS

Closed-source, high-end LLMs exhibit superior performance in JRBE, and translating questions into English by themselves further enhances their accuracy. There is a significant positive correlation between the cost of LLMs and their performance, whereas the release date does not significantly influence their performance.