Automated Quality Control of Breast Ultrasound Reports Using a BI-RADS-Prompted Large Language Model: A Pilot Multicenter Study Involving 60 Hospitals in China.

OBJECTIVE

To evaluate the feasibility of using large language model (LLM) for automated quality control (QC) of breast ultrasound (US) reports.

METHODS

In this retrospective multicenter study, we collected breast US reports from 735 patients who had pathological confirmation of mass-type lesions on breast US across 60 hospitals in China. Each hospital's QC personnel converted free-text reports into standardized structured outputs. A gold standard was established through a multilevel expert review. The Qwen2.5-VL-7B LLM was applied to the same free-text reports, generating structured reports based on BI-RADS prompts. Accuracy was compared both between the LLM and QC personnel as well as the times required to produce the outputs for analysis of efficiency.

RESULTS

The LLM demonstrated higher accuracy in conducting QC for key breast lesion US features, such as margin (80.0% versus 64.1%, P < .0001) and echo pattern (74.0% versus 56.1%, P < .0001). Subgroup analysis further confirmed its robustness: in the complex reports of multiple lesions, it maintained its advantage in margin QC (79.8% versus 64.4%). The study also revealed a significant positive correlation between the LLM's QC accuracy and BI-RADS categories (from 3 to 5) (Spearman ρ = 0.264, P = .028), a trend not observed in manual QC. In terms of efficiency, the LLM completed QC for 50 reports in an average of only 13 min, faster than 212.5 min required by manual reviewers.

CONCLUSIONS

The proposed LLM-based system provides a reliable, accurate, and efficient solution for breast US reports. It achieves human-comparable performance with markedly higher efficiency, particularly in reports with high suspicion levels, offering a reliable tool to enhance report quality.