Performance of large language models in complex coronary artery disease decision-making and potential improvement strategy in a real-world cohort.

OBJECTIVE

Previous studies have demonstrated that large language models (LLMs) can achieve performance comparable to multidisciplinary heart teams (HTs) in coronary revascularisation decision-making but limited by population bias. We aimed to assess LLMs’ performance in complex coronary artery disease (CAD) decision-making using a real-world cohort and to explore the potential improvement strategy.

METHODS

Patients with stable CAD and angiographically confirmed three-vessel or left main disease randomly selected from a consecutive angiography registry study were included in this cross-sectional study. ChatGPT-4.1 was used to provide five-category treatment decisions based on inputted patient information and a prompt most commonly used by real-world patients. The primary outcome was the agreement rate between HT and LLM recommendations. Three predefined prompting strategies were further evaluated for their potential to enhance agreement rates.

RESULTS

A total of 564 patients were included. The agreement rate between HT and LLM recommendations was 47.0% based on initial prompt. LLM performed poorly in further testing (sensitivity=0.0%) decision. LLM performed well in patients with Synergy between Percutaneous Coronary Intervention with Taxus and Cardiac Surgery score ≥33 (94.1%) but poorly in 23–32 (61.7%) and 0–22 (20.9%). After using prompting strategies, the agreement rate increased significantly (79.3%, p<0.001).

DISCUSSION: Simple patient-style queries cannot independently generate reliable decisions for real-world complex revascularisation scenarios. While prompt engineering and subgroup analysis can partially improve LLM’s performance, the potential safety risk remains emphasised.

CONCLUSION

Among a real-world cohort of patients with complex CAD, the agreement rate of decision-making between LLM and HTs was only half, but could be improved with appropriate prompts.