Evaluation of large language models for TI-RADS category extraction from free-text thyroid US reports: a comparative study with human readers.

OBJECTIVE: To evaluate the performance of two large language models (LLMs) in extracting and assigning the American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) categories from free-text thyroid ultrasound (US) reports. METHODS: This retrospective study included thyroid US reports from patients who underwent thyroid examinations. Three experienced US radiologists (with > 15 years of experience) and three general radiologists (with 4–5 years of experience) reviewed these free-text reports and assigned an ACR TI-RADS category based on the original US descriptions. The same reports were processed using standardized prompts in GPT-4.1 and Gemini-2.5-pro. Inter-rater agreement was evaluated using Gwet’s AC1 coefficient. RESULTS: Inter-rater agreement between human readers was moderate (Gwet’s AC1 = 0.64, 95% CI 0.62–0.68) and was significantly higher than that between two LLMs (Gwet’s AC1 = 0.47, 95% CI 0.44–0.50; P < 0.001). Agreement between expert radiologists and the LLMs was substantial (Gwet’s AC1 = 0.67, 95% CI 0.65–0.70 for GPT-4.1 and 0.66, 95% CI 0.63–0.69 for Gemini-2.5-pro), whereas agreement with general readers was lower (Gwet’s AC1 = 0.49, 95% CI 0.46–0.52 for GPT-4.1, and 0.47, 95% CI 0.44–0.50 for Gemini-2.5-pro). The AUCs were 0.859 (95% CI, 0.831–0.887) for expert radiologists, 0.827 (95% CI, 0.798–0.856) for general radiologists, 0.836 (95% CI, 0.813–0.859) for Gemini-2.5-pro, and 0.811 (95% CI, 0.782–0.839) for GPT-4.1. CONCLUSION: LLMs demonstrated promising performance in assigning TI-RADS category from free-text thyroid US reports, achieving accuracy and agreement comparable to that of general radiologists. However, they are not yet ready to replace expert interpretation in high-stakes clinical settings.