Do question type and topic affect the performance of large language models in answering periodontology questions? A comparative study.

BACKGROUND

There is increasing interest and ongoing debate about the integration of artificial intelligence and the large language models it supports in health science education. The performance of these models may vary depending on educational domains, clinical branches, complexity of questions, and question formats. This study aimed to compare the performance of ChatGPT-5, Gemini 2.5 Flash, and Copilot on periodontology questions and to determine whether their performance was affected by question type and topic.

METHODS

One hundred thirty-four multiple-choice periodontology questions from the dental specialty exam in Türkiye were fed to these models. The responses were evaluated based on both their accuracy and the quality of their explanations. A Likert scale was used to examine the quality of explanation. The answers from the models were analyzed comparatively based on question type and topic. Regression analyses were performed to determine the effect of question type and topic.

RESULTS

The total accuracy rates of ChatGPT-5, Gemini 2.5 Flash, and Copilot were similar. Overall, Gemini 2.5 Flash outperformed the others in terms of adequacy of response explanation (p < 0.05). Gemini provided more consistent explanations than others on basic science and clinical periodontology questions, as well as standardized and information-based questions (p < 0.05, for all). Regression analysis showed that question type and topic were related to explanation quality.

CONCLUSIONS

According to the findings of this study, question types and topics seem to affect the consistency and quality of large language models' explanations more than the accuracy of the answers. This suggests that large language models are promising for periodontology education, but they still have limitations that require improvement. However, these findings should be interpreted cautiously within the limitations of the study.