World neurosurgery · 2026
Assessing Artificial Intelligence Language Models for Patient-Oriented Information on Chiari Malformation Types: A Structured Multinational Evaluation Based on Real-World Expectations.
СтатьяНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Çetin Mert, Turgut Ali Çağlar, Beger Orhan, Wang Fang, Dumont Aaron S, Guo Fuyou, Turgut Mehmet, Tubbs R Shane
Аннотация
OBJECTIVE: To evaluate the performance of contemporary large language model (LLM)-based artificial intelligence (AI) systems in providing patient-oriented information across multiple Chiari malformation (CM) subtypes.
METHODS: Five AI models (ChatGPT-4o, Gemini, Copilot, DeepSeek, and Perplexity AI) were evaluated using seven patient-oriented questions on the definition, epidemiology, etiology, symptomatology, diagnosis, treatment, and prognosis of CM. Nine CM subtypes were included: Types 0, 0.5, 1, 1.5, 2, 3, 3.5, 4, and 5. A total of 315 AI-generated responses were independently assessed by three blinded neurosurgeons using a three-point ordinal scoring system for accuracy, comprehensiveness, and conciseness.
RESULTS: There were significant differences among the AI models across all domains evaluated (all p<0.001). Gemini performed most strongly in accuracy and comprehensiveness, while Perplexity performed best in conciseness. Copilot generally performed worse across the domains evaluated. Significant subtype-based differences were also identified, with AI-generated responses concerning Types 1 and 2 performing substantially better than those concerning rarer subtypes. Diagnostic questions achieved the highest overall performance, while prognosis- and definition-related questions performed comparatively less well. Reviewer-based analyses revealed substantial variability in scoring behavior.
CONCLUSIONS: Contemporary LLMs perform variably in providing patient-oriented information about CMs. Although some AI systems generated relatively accurate and comprehensive responses for well-established CM subtypes, performance was lower for rarer and less clearly defined variants, highlighting the continuing importance of expert oversight in delivering AI-assisted medical information.
METHODS: Five AI models (ChatGPT-4o, Gemini, Copilot, DeepSeek, and Perplexity AI) were evaluated using seven patient-oriented questions on the definition, epidemiology, etiology, symptomatology, diagnosis, treatment, and prognosis of CM. Nine CM subtypes were included: Types 0, 0.5, 1, 1.5, 2, 3, 3.5, 4, and 5. A total of 315 AI-generated responses were independently assessed by three blinded neurosurgeons using a three-point ordinal scoring system for accuracy, comprehensiveness, and conciseness.
RESULTS: There were significant differences among the AI models across all domains evaluated (all p<0.001). Gemini performed most strongly in accuracy and comprehensiveness, while Perplexity performed best in conciseness. Copilot generally performed worse across the domains evaluated. Significant subtype-based differences were also identified, with AI-generated responses concerning Types 1 and 2 performing substantially better than those concerning rarer subtypes. Diagnostic questions achieved the highest overall performance, while prognosis- and definition-related questions performed comparatively less well. Reviewer-based analyses revealed substantial variability in scoring behavior.
CONCLUSIONS: Contemporary LLMs perform variably in providing patient-oriented information about CMs. Although some AI systems generated relatively accurate and comprehensive responses for well-established CM subtypes, performance was lower for rarer and less clearly defined variants, highlighting the continuing importance of expert oversight in delivering AI-assisted medical information.