Journal of dental sciences · 2025
Analysis of multimodal large language models on visually-based questions in the Japanese National Examination for Dental Hygienists: A preliminary comparative study.
СтатьяНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Kaneyasu Yoshino, Mine Yuichi, Niitani Yoshie, Taji Tsuyoshi, Takeda Saori, Tokinaga Ryohei, Shigeishi Hideo, Takemoto Toshinobu, Kakimoto Naoya, Murayama Takeshi, Ohta Kouji
Аннотация
BACKGROUND/PURPOSE: The importance of oral health is globally recognized, which has increased the demand for qualified dental hygienists. This study assessed the performance of multimodal large language models (LLMs), on the Japanese National Examination for Dental Hygienists, focusing on their ability to answer visually-based questions and evaluating image-recognition capabilities.
MATERIALS AND METHODS: The 34th Japanese National Examination for Dental Hygienists (March 2025) supplied 213 multiple-choice questions (74 text-only, 139 visually-based). Five multimodal LLMs were tested: OpenAI o3-mini-high (o3-mh), ChatGPT-4.5 Preview (GPT-4.5), Gemini 2.0 Flash Thinking Experimental (Gemini 2.0), Gemini 2.5 Pro Experimental (Gemini 2.5), and Claude 3.7 Sonnet (Claude 3.7). Performance was evaluated by comparing LLM answers to official correct answers. Cochran's Q test and McNemar's tests with Bonferroni correction were used for statistical analysis.
RESULTS: Gemini 2.5 achieved the highest overall correct response rate (85.0 %), followed by Claude 3.7 (77.5 %), o3-mh (77.0 %), GPT-4.5 (76.1 %), and Gemini 2.0 (75.1 %). For text-only questions, Claude 3.7 (91.9 %) performed best. On visually-based questions, Gemini 2.5 was superior (82.0 %), while other models scored around 70-73 %. Gemini 2.5 significantly outperformed, GPT-4.5 and Gemini 2.0 overall, and GPT-4.5 and Claude 3.7 on visually-based questions.
CONCLUSION: Multimodal LLMs, particularly Gemini 2.5, demonstrate significant proficiency on the Japanese National Examination for Dental Hygienists, including questions with visual elements. These findings suggest a growing potential for LLMs as educational tools in dental hygiene. However, current limitations in accuracy and reliability necessitate further refinement and cautious integration into educational and clinical settings.
MATERIALS AND METHODS: The 34th Japanese National Examination for Dental Hygienists (March 2025) supplied 213 multiple-choice questions (74 text-only, 139 visually-based). Five multimodal LLMs were tested: OpenAI o3-mini-high (o3-mh), ChatGPT-4.5 Preview (GPT-4.5), Gemini 2.0 Flash Thinking Experimental (Gemini 2.0), Gemini 2.5 Pro Experimental (Gemini 2.5), and Claude 3.7 Sonnet (Claude 3.7). Performance was evaluated by comparing LLM answers to official correct answers. Cochran's Q test and McNemar's tests with Bonferroni correction were used for statistical analysis.
RESULTS: Gemini 2.5 achieved the highest overall correct response rate (85.0 %), followed by Claude 3.7 (77.5 %), o3-mh (77.0 %), GPT-4.5 (76.1 %), and Gemini 2.0 (75.1 %). For text-only questions, Claude 3.7 (91.9 %) performed best. On visually-based questions, Gemini 2.5 was superior (82.0 %), while other models scored around 70-73 %. Gemini 2.5 significantly outperformed, GPT-4.5 and Gemini 2.0 overall, and GPT-4.5 and Claude 3.7 on visually-based questions.
CONCLUSION: Multimodal LLMs, particularly Gemini 2.5, demonstrate significant proficiency on the Japanese National Examination for Dental Hygienists, including questions with visual elements. These findings suggest a growing potential for LLMs as educational tools in dental hygiene. However, current limitations in accuracy and reliability necessitate further refinement and cautious integration into educational and clinical settings.