Orbit (Amsterdam, Netherlands) · 2025
Evaluating large language models in answering patient questions about eye removal surgeries.
СтатьяНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Bineshfar Niloufar, Shields Chloe, Davila Natalia, Panneerselvam Sugi, Pradeep Tejus, Shoji Marissa K, Lee Wendy W
Аннотация
PURPOSE: To evaluate the performance of ChatGPT-4 and Gemini, two large language models (LLMs), in addressing frequently asked questions (FAQs) about eye removal surgeries.
METHODS: A set of 24 FAQs related to enucleation and evisceration was identified through a Google search and categorized into preoperative, procedural, and postoperative topics. Each question was submitted three times to ChatGPT-4o and Gemini, and responses were evaluated for consistency, accuracy, appropriateness, and potential harm. Readability was assessed using Flesch Reading Ease and Flesch-Kincaid Grade Level scores.
RESULTS: Gemini exhibited higher response consistency compared to ChatGPT (p = 0.043), while ChatGPT produced longer responses (mean length: 169.3 vs. 109.9 words; p < 0.001). Gemini's responses were more readable, with a higher Flesch Reading Ease score (39.0 vs. 31.3, p = 0.001) and lower Flesch-Kincaid Grade Level (11.6 vs. 14.0, p < 0.001). Both LLMs demonstrated comparable accuracy and low potential for harm, with 79.2% of Gemini responses and 77.1% of ChatGPT responses rated as completely correct. The sources cited by Gemini included academic institutions (91.7%) and medical practices (8.3%), while ChatGPT exclusively referenced academic sources.
CONCLUSIONS: ChatGPT and Gemini showed comparable accuracy and low harm potential when addressing patient questions about eye removal surgeries. Gemini provided more consistent and readable responses, but both LLMs exceeded the recommended readability levels for patient education. These findings highlight the potential of LLMs to assist in patient communication and clinical education while underscoring the need for careful oversight in their implementation.
METHODS: A set of 24 FAQs related to enucleation and evisceration was identified through a Google search and categorized into preoperative, procedural, and postoperative topics. Each question was submitted three times to ChatGPT-4o and Gemini, and responses were evaluated for consistency, accuracy, appropriateness, and potential harm. Readability was assessed using Flesch Reading Ease and Flesch-Kincaid Grade Level scores.
RESULTS: Gemini exhibited higher response consistency compared to ChatGPT (p = 0.043), while ChatGPT produced longer responses (mean length: 169.3 vs. 109.9 words; p < 0.001). Gemini's responses were more readable, with a higher Flesch Reading Ease score (39.0 vs. 31.3, p = 0.001) and lower Flesch-Kincaid Grade Level (11.6 vs. 14.0, p < 0.001). Both LLMs demonstrated comparable accuracy and low potential for harm, with 79.2% of Gemini responses and 77.1% of ChatGPT responses rated as completely correct. The sources cited by Gemini included academic institutions (91.7%) and medical practices (8.3%), while ChatGPT exclusively referenced academic sources.
CONCLUSIONS: ChatGPT and Gemini showed comparable accuracy and low harm potential when addressing patient questions about eye removal surgeries. Gemini provided more consistent and readable responses, but both LLMs exceeded the recommended readability levels for patient education. These findings highlight the potential of LLMs to assist in patient communication and clinical education while underscoring the need for careful oversight in their implementation.