A Comparative Study: Can Large Language Models be a Supportive Tool in the Diagnosis and Treatment of Scabies?

BACKGROUND

The advent of large language models (LLMs) has endowed artificial intelligence with remarkable capabilities to comprehend and generate human-like text by leveraging extensive datasets. Numerous studies in medicine increasingly focus on LLMs, particularly highlighting their substantial potential in diagnostic and therapeutic management. Scabies is a contagious skin disease that continues to affect over 200 million people globally each year and is particularly prevalent in developing countries. In recent years, there has been a marked increase in scabies cases both in Turkey and worldwide, elevating scabies to the status of a public health concern. Therefore, the Clinical Practice Guideline for the Diagnosis and Treatment of Scabies (CPGDT) was published in 2024.

AIMS AND

OBJECTIVES

This study evaluates the performance of LLMs on scabies-related questions created from CPGDT and compares it to a dermatologist.

MATERIALS AND

METHODS

Sixty-four multiple-choice questions in English were administered to nine LLMs, including Claude 3 Opus, ChatGPT (versions 3.5, 4, and 4o), Google Gemini (1.0 and 1.5 Pro), Microsoft Copilot, Meta LLaMA 3 70B, and Perplexity, as well as to a senior dermatologist with 22 years of experience in dermatology. Responses were scored as correct or incorrect. Statistical analyses, including the McNemar test, were conducted using IBM SPSS Statistics 23.0.

RESULTS

Claude 3 Opus achieved the highest accuracy (89%), significantly outperforming other LLMs and dermatologists (75%) (P < 0.05). ChatGPT 4o (79.6%) and ChatGPT 4 (78.1%) followed, with no significant differences observed among other LLMs or between LLMs and dermatologists (P > 0.05).

CONCLUSION

This is the first study evaluating LLMs' performance on scabies-related knowledge. Claude 3 Opus' superior performance highlights its potential in dermatology, surpassing widely recognized LLMs like ChatGPT. Claude 3 Opus demonstrated exceptional accuracy in scabies-related queries, underscoring its value as adjunctive tool in dermatology. Enhancing LLMs' visual diagnostic capabilities is vital for their integration into clinical practice.