Frontiers in public health · 2026
Performance comparison of large language models for medication counseling in people living with HIV.
СтатьяСравнительное исследованиеНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Huang Can, Sun Yanfang, Chen Meng, Zhang Lin, Liu Wei
Аннотация
OBJECTIVE: This study aimed to evaluate the comprehensive performance of five large language models (LLMs), namely ChatGPT, DeepSeek, Doubao, Kimi, and Qwen, in addressing medication consultation inquiries for people living with HIV (PLWH) in a Chinese-language context, thereby providing evidence for their clinical application and further model optimization.
METHODS: A total of 55 real-world medication consultation questions covering mainstream antiretroviral drugs for PLWH were screened and classified from Beijing Youan Hospital, Capital Medical University, a specialized infectious disease hospital in China. Five LLMs were queried within a fixed period, and expert evaluations were conducted across five dimensions: accuracy, relevance, completeness, clarity, and reliability.
RESULTS: The comprehensive scores ranked from highest to lowest were DeepSeek (4.47), Qwen (4.33), Kimi (4.24), Doubao (4.13), and ChatGPT (3.41), with highly significant differences were observed among all models (H=182.14, p < 0.001). Regarding dimensional scores, the ranking was clarity (4.53) > reliability (4.33) > completeness (4.13) > accuracy (3.95) > relevance (3.62). ChatGPT exhibited statistically significant differences compared with all other models (p < 0.001); no significant differences were found between DeepSeek and Qwen or between Doubao and Kimi (p > 0.05).
CONCLUSION: Significant differences were observed in the capacity of the five LLMs to address medication consultations for PLWH within the Chinese-language context. DeepSeek and Qwen achieved optimal overall performance, Doubao excelled in clarity, whereas ChatGPT yielded the poorest results. All models demonstrated significant limitations when handling complex pharmaceutical inquiries and cannot fully replace professional clinical pharmacists. Further optimization focusing on high-quality medical domain dataset training and algorithm refinement is therefore warranted.
METHODS: A total of 55 real-world medication consultation questions covering mainstream antiretroviral drugs for PLWH were screened and classified from Beijing Youan Hospital, Capital Medical University, a specialized infectious disease hospital in China. Five LLMs were queried within a fixed period, and expert evaluations were conducted across five dimensions: accuracy, relevance, completeness, clarity, and reliability.
RESULTS: The comprehensive scores ranked from highest to lowest were DeepSeek (4.47), Qwen (4.33), Kimi (4.24), Doubao (4.13), and ChatGPT (3.41), with highly significant differences were observed among all models (H=182.14, p < 0.001). Regarding dimensional scores, the ranking was clarity (4.53) > reliability (4.33) > completeness (4.13) > accuracy (3.95) > relevance (3.62). ChatGPT exhibited statistically significant differences compared with all other models (p < 0.001); no significant differences were found between DeepSeek and Qwen or between Doubao and Kimi (p > 0.05).
CONCLUSION: Significant differences were observed in the capacity of the five LLMs to address medication consultations for PLWH within the Chinese-language context. DeepSeek and Qwen achieved optimal overall performance, Doubao excelled in clarity, whereas ChatGPT yielded the poorest results. All models demonstrated significant limitations when handling complex pharmaceutical inquiries and cannot fully replace professional clinical pharmacists. Further optimization focusing on high-quality medical domain dataset training and algorithm refinement is therefore warranted.