Journal of dental sciences · 2025
Evaluation of GPT-4o and Gemini Advanced on the Korean National Dental Licensing Examination: Accuracy, consistency, and question generation.
СтатьяНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Song Eun Sun, Kim Ga Hyeon, Lee Seung-Pyo
Аннотация
BACKGROUND/PURPOSE: Large language models (LLMs), such as GPT-4o and Gemini Advanced, have performed strongly on global medical examinations. However, their capabilities in non-English, dentistry-specific licensing contexts remain unclear. Thus, this study aimed to compare the performance, consistency, and question-generation abilities of GPT-4o and Gemini Advanced in the Korean National Dental Licensing Examination (KNDLE).
MATERIALS AND METHODS: This study used 1,401 text-based KNDLE questions from 2019 to 2023 in Korean. Each model responded to the questions in three separate runs. Accuracy and consistency were compared with human answers. The models generated new questions in four subject areas and attempted to solve each other's generated items. Paired t-tests and chi-square tests were conducted.
RESULTS: GPT-4o achieved significantly higher average accuracy than Gemini Advanced (81.1 % vs. 76.6 %, P = 0.013) and showed greater consistency across attempts. Both models performed better in basic sciences than in clinical subjects, such as prosthodontics. In cross-solving tasks, GPT-4o's performance notably declined in Gemini-generated oral biology questions, indicating interpretation differences. However, the consistency difference between models was not significant (P = 0.578).
CONCLUSION: GPT-4o outperformed Gemini Advanced in accuracy, consistency, and alignment with its generated content. However, challenges remain in clinical domains and cross-model understanding, highlighting the potential of LLMs as supportive tools for non-English dental education and question generation while emphasizing the persistent need for expert oversight and domain-specific refinement.
MATERIALS AND METHODS: This study used 1,401 text-based KNDLE questions from 2019 to 2023 in Korean. Each model responded to the questions in three separate runs. Accuracy and consistency were compared with human answers. The models generated new questions in four subject areas and attempted to solve each other's generated items. Paired t-tests and chi-square tests were conducted.
RESULTS: GPT-4o achieved significantly higher average accuracy than Gemini Advanced (81.1 % vs. 76.6 %, P = 0.013) and showed greater consistency across attempts. Both models performed better in basic sciences than in clinical subjects, such as prosthodontics. In cross-solving tasks, GPT-4o's performance notably declined in Gemini-generated oral biology questions, indicating interpretation differences. However, the consistency difference between models was not significant (P = 0.578).
CONCLUSION: GPT-4o outperformed Gemini Advanced in accuracy, consistency, and alignment with its generated content. However, challenges remain in clinical domains and cross-model understanding, highlighting the potential of LLMs as supportive tools for non-English dental education and question generation while emphasizing the persistent need for expert oversight and domain-specific refinement.