Evaluation of large language models for TI-RADS category extraction from free-text thyroid US reports: a comparative study with human readers.

OBJECTIVE

To evaluate the performance of two large language models (LLMs) in extracting and assigning the American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) categories from free-text thyroid ultrasound (US) reports.

METHODS

This retrospective study included thyroid US reports from patients who underwent thyroid examinations. Three experienced US radiologists (with > 15 years of experience) and three general radiologists (with 4–5 years of experience) reviewed these free-text reports and assigned an ACR TI-RADS category based on the original US descriptions. The same reports were processed using standardized prompts in GPT-4.1 and Gemini-2.5-pro. Inter-rater agreement was evaluated using Gwet’s AC1 coefficient.

RESULTS

Inter-rater agreement between human readers was moderate (Gwet’s AC1 = 0.64, 95% CI 0.62–0.68) and was significantly higher than that between two LLMs (Gwet’s AC1 = 0.47, 95% CI 0.44–0.50; P < 0.001). Agreement between expert radiologists and the LLMs was substantial (Gwet’s AC1 = 0.67, 95% CI 0.65–0.70 for GPT-4.1 and 0.66, 95% CI 0.63–0.69 for Gemini-2.5-pro), whereas agreement with general readers was lower (Gwet’s AC1 = 0.49, 95% CI 0.46–0.52 for GPT-4.1, and 0.47, 95% CI 0.44–0.50 for Gemini-2.5-pro). The AUCs were 0.859 (95% CI, 0.831–0.887) for expert radiologists, 0.827 (95% CI, 0.798–0.856) for general radiologists, 0.836 (95% CI, 0.813–0.859) for Gemini-2.5-pro, and 0.811 (95% CI, 0.782–0.839) for GPT-4.1.

CONCLUSION

LLMs demonstrated promising performance in assigning TI-RADS category from free-text thyroid US reports, achieving accuracy and agreement comparable to that of general radiologists. However, they are not yet ready to replace expert interpretation in high-stakes clinical settings.