Performance of large language models in neonatal resuscitation assessments versus healthcare providers: an exploratory study.

BACKGROUND

Artificial intelligence and large language models (LLMs) have developed rapidly in recent years and involved in medical education, in addition to clinical care. However, it remains unknown how the performance of LLMs in neonatal resuscitation compares to that of healthcare professionals (HCPs). In this exploratory study, we aimed to investigate and compare the performance of LLMs with those of HCPs on 3 sources of examination questions in the neonatal resuscitation training in Canada and China.

METHODS

In this bi-center study, we evaluated the overall accuracy, accuracy across question types, and reliability of LLMs' (ChatGPT-5 and DeepSeek-R1) responses to neonatal resuscitation questions from workshop in China, NRP® textbook (8th edition), and Kahoot quizzes in Canada. Each LLM was tested three times per question with one-week intervals in August and September, 2025. Comparisons with historical scores of HCPs in the training courses in China (2022-2024) and Canada (2022-2025) were performed.

RESULTS

Both LLMs performed comparably to HCPs in Chinese examinations, and showed similar accuracy in NRP® textbook questions with higher scores on multiple-choice than on short answer questions. ChatGPT achieved higher accuracy than DeepSeek and HCPs in the Kahoot quizzes. ChatGPT also had higher accuracy on scenario-based than on non-scenario-based questions in the workshop examination. High reliability of LLMs' responses was found (Fleiss' Kappa>0.89).

CONCLUSION

Both ChatGPT and DeepSeek achieved accuracy comparable to that of HCPs and showed high consistency on selected neonatal resuscitation written examinations. The findings warrant further research to explore their potential integration in neonatal resuscitation training.