Performance evaluation of 9 large language models in anticoagulation decision-making for nonvalvular atrial fibrillation.

BACKGROUND

Large language models (LLMs) are increasingly applied in clinical decision support, but their reliability in guiding anticoagulation management for nonvalvular atrial fibrillation (NVAF) remains unclear.

OBJECTIVES

To systematically evaluate the performance of 9 widely used LLMs in guiding anticoagulation therapy decisions for NVAF.

METHODS

A multidisciplinary team constructed 100 virtual NVAF patient cases with comprehensive demographic, clinical, laboratory, and medication information. Nine LLMs (DeepSeek-V3, DeepSeek-R1, Qwen3, Qwen3 Deep Think, ChatGPT-4o, ChatGPT-4o Deep Research, Gemini 2.5, Gemini 2.5 Deep Research, and Claude Sonnet 4) were prompted to provide individualized Congestive heart failure, Hypertension, Age ≥75 years, Diabetes mellitus, Stroke, Vascular disease, Age 65-74 years, Sex category (CHA2DS2-VASc) and Hypertension, Abnormal Renal/Liver Function, Stroke, Bleeding History or Predisposition, Labile INR, Elderly, Drugs/Alcohol Concomitantly (HAS-BLED) scores, anticoagulation contraindications, anticoagulation decisions, regimens, and monitoring recommendations. Outputs were independently evaluated against the multidisciplinary team consensus using accuracy rates and Likert-scale scores, with statistical comparisons performed across models.

RESULTS

Across 900 responses, performance varied substantially across 6 clinical tasks. ChatGPT-4o Deep Research achieved the highest accuracy in CHA2DS2-VASc scoring (100%) and medication monitoring (mean score, 4.98), ranking first overall. Claude Sonnet 4 demonstrated the best performance in anticoagulation regimen selection (89% accuracy), ranking second, while DeepSeek-V3 showed the lowest overall performance. Across all models, HAS-BLED scoring accuracy was consistently low (8%-51%). Notably, prompt optimization markedly improved HAS-BLED scoring accuracy, reaching 95% with ChatGPT-4o Deep Research and 89% with DeepSeek-R1 (P < .001).

CONCLUSION

There was significant variability among LLMs in their decision support for NVAF anticoagulation. Prompt optimization has the potential to markedly enhance accuracy. At present, LLMs may serve as valuable adjuncts to clinical practice but remain unsuitable for use as stand-alone decision-makers.