International journal of medical informatics · 2026
Accuracy and completeness of large language models in Epidemic keratoconjunctivitis Queries: A Comparative study.
СтатьяСравнительное исследованиеНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Es'haghi Acieh, Aliyariparand Mohsen, Jamalipour Soufi Kaveh, Aghaei Hossein
Аннотация
BACKGROUND: Large language models (LLMs) are increasingly applied in clinical contexts, yet their reliability in disease-specific ophthalmic domains remains insufficiently characterized. Epidemic keratoconjunctivitis (EKC), a highly contagious adenoviral ocular infection, represents a critical area where accurate diagnostic and therapeutic information is essential. This study aimed to assess and compare the accuracy and completeness of responses generated by two widely used LLMs, namely ChatGPT and Meta AI, to standardized ophthalmology questions related to EKC.
METHODS: This cross-sectional expert evaluation study was conducted between January and March 2025. Thirty-four structured questions covering five EKC domains (etiology, clinical features, diagnosis, treatment, and prognosis) were developed using authoritative ophthalmology sources. Responses generated by ChatGPT (Plus subscription, web interface) and Meta AI (web interface) were independently scored by two fellowship-trained cornea specialists using a five-point Likert scale for accuracy and completeness. Inter-rater agreement was assessed using Cohen's κ and mean absolute error (MAE).
RESULTS: Both models demonstrated strong overall performance, with mean accuracy and completeness scores of 4.87 and 4.90 for ChatGPT and 4.87 and 4.93 for Meta AI, respectively. Agreement between expert raters was substantial (κ range 0.78-0.88). Domain-specific analysis revealed consistently high scores across all categories, with Meta AI showing slightly higher completeness, particularly in diagnostic and symptom-related responses.
CONCLUSION: LLMs exhibit consistently strong performance and domain consistency when addressing ophthalmic queries related to EKC. Their performance supports their potential use as adjunct informational and educational tools for ophthalmic conditions such as EKC, particularly for structured knowledge retrieval, treatment explanation, and patient counseling, provided that use remains under appropriate professional oversight.
METHODS: This cross-sectional expert evaluation study was conducted between January and March 2025. Thirty-four structured questions covering five EKC domains (etiology, clinical features, diagnosis, treatment, and prognosis) were developed using authoritative ophthalmology sources. Responses generated by ChatGPT (Plus subscription, web interface) and Meta AI (web interface) were independently scored by two fellowship-trained cornea specialists using a five-point Likert scale for accuracy and completeness. Inter-rater agreement was assessed using Cohen's κ and mean absolute error (MAE).
RESULTS: Both models demonstrated strong overall performance, with mean accuracy and completeness scores of 4.87 and 4.90 for ChatGPT and 4.87 and 4.93 for Meta AI, respectively. Agreement between expert raters was substantial (κ range 0.78-0.88). Domain-specific analysis revealed consistently high scores across all categories, with Meta AI showing slightly higher completeness, particularly in diagnostic and symptom-related responses.
CONCLUSION: LLMs exhibit consistently strong performance and domain consistency when addressing ophthalmic queries related to EKC. Their performance supports their potential use as adjunct informational and educational tools for ophthalmic conditions such as EKC, particularly for structured knowledge retrieval, treatment explanation, and patient counseling, provided that use remains under appropriate professional oversight.