Performance comparison of large language models for medication counseling in people living with HIV.

OBJECTIVE

This study aimed to evaluate the comprehensive performance of five large language models (LLMs), namely ChatGPT, DeepSeek, Doubao, Kimi, and Qwen, in addressing medication consultation inquiries for people living with HIV (PLWH) in a Chinese-language context, thereby providing evidence for their clinical application and further model optimization.

METHODS

A total of 55 real-world medication consultation questions covering mainstream antiretroviral drugs for PLWH were screened and classified from Beijing Youan Hospital, Capital Medical University, a specialized infectious disease hospital in China. Five LLMs were queried within a fixed period, and expert evaluations were conducted across five dimensions: accuracy, relevance, completeness, clarity, and reliability.

RESULTS

The comprehensive scores ranked from highest to lowest were DeepSeek (4.47), Qwen (4.33), Kimi (4.24), Doubao (4.13), and ChatGPT (3.41), with highly significant differences were observed among all models (H=182.14, p < 0.001). Regarding dimensional scores, the ranking was clarity (4.53) > reliability (4.33) > completeness (4.13) > accuracy (3.95) > relevance (3.62). ChatGPT exhibited statistically significant differences compared with all other models (p < 0.001); no significant differences were found between DeepSeek and Qwen or between Doubao and Kimi (p > 0.05).

CONCLUSION

Significant differences were observed in the capacity of the five LLMs to address medication consultations for PLWH within the Chinese-language context. DeepSeek and Qwen achieved optimal overall performance, Doubao excelled in clarity, whereas ChatGPT yielded the poorest results. All models demonstrated significant limitations when handling complex pharmaceutical inquiries and cannot fully replace professional clinical pharmacists. Further optimization focusing on high-quality medical domain dataset training and algorithm refinement is therefore warranted.