Kardiochirurgia i torakochirurgia polska = Polish journal of cardio-thoracic surgery · 2025
ChatGPT-3.5 and the Polish thoracic surgery specialty examination: a performance evaluation.
СтатьяНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Mitręga Adam, Kaczyńska Dominika, Magiera Mikołaj, Denisiewicz Natalia, Bielówka Michał, Kożuch Anna, Korbaś Miłosz, Gaweł Aleksandra, Kufel Jakub
Аннотация
INTRODUCTION: The incredibly rapid development of artificial intelligence (AI) in recent years has created new opportunities for its application in medical advancements. This raises questions about the reliability and limitations of AI.
AIM: The aim of the present study was to evaluate the effectiveness of the ChatGPT-3.5 language model in solving the test component of the National Specialist Examination (PES) in the field of thoracic surgery.
MATERIAL AND METHODS: A total of 120 test questions from 2015 PES examination were analyzed. They were grouped according to subject matter, clinical character, and cognitive requirements. In independent sessions, each question was submitted five times. The following statistical tests were applied: χ2, Kruskal-Wallis, Mann-Whitney and Spearman's rank correlation. The consistency of the answers was assessed using Fleiss' k coefficient.
RESULTS: The AI tool achieved a score of 42.2% correct answers, with the passing threshold set at 60%. A statistically significant difference was found between clinical and non-clinical questions (p = 0.041). Correct answers were characterized by a higher confidence coefficient (p < 0.001). No correlation was observed between confidence and psychometric indicators. The response consistency was assessed as moderate (k = 0.341).
CONCLUSIONS: The result obtained by ChatGPT-3.5 is equivalent to a failing score on the examination. The confidence of responses correlated with their correctness, whereas limitations in clinical knowledge and consistency indicate the need for caution when using this model to assess specialized knowledge.
AIM: The aim of the present study was to evaluate the effectiveness of the ChatGPT-3.5 language model in solving the test component of the National Specialist Examination (PES) in the field of thoracic surgery.
MATERIAL AND METHODS: A total of 120 test questions from 2015 PES examination were analyzed. They were grouped according to subject matter, clinical character, and cognitive requirements. In independent sessions, each question was submitted five times. The following statistical tests were applied: χ2, Kruskal-Wallis, Mann-Whitney and Spearman's rank correlation. The consistency of the answers was assessed using Fleiss' k coefficient.
RESULTS: The AI tool achieved a score of 42.2% correct answers, with the passing threshold set at 60%. A statistically significant difference was found between clinical and non-clinical questions (p = 0.041). Correct answers were characterized by a higher confidence coefficient (p < 0.001). No correlation was observed between confidence and psychometric indicators. The response consistency was assessed as moderate (k = 0.341).
CONCLUSIONS: The result obtained by ChatGPT-3.5 is equivalent to a failing score on the examination. The confidence of responses correlated with their correctness, whereas limitations in clinical knowledge and consistency indicate the need for caution when using this model to assess specialized knowledge.