Academic radiology · 2026
Comparative Evaluation of Open-Weight Large Language Models for Pathological Brain MRI Reports Classification: From Prompts to a Fine-Tuned Model.
СтатьяНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Demaria Claudio, Crema Claudio, Verdicchio Mario, Brancato Valentina, Alfano Vincenzo, Basso Luca, Pontillo Giuseppe, Archetti Damiano, Fostinelli Silvia, De Francesco Silvia, Pievani Michela, Baronio Cesare M, Boccali Alberto, Toschi Nicola, Brunetti Arturo, Redolfi Alberto, Aiello Marco
Аннотация
RATIONALE AND OBJECTIVES: Automated semantic classification of radiology reports can enhance clinical workflows and research, but often relies on language- and task-specific models, limiting scalability. This study aims to evaluate and improve the 0-shot baseline performance of open-weight large language models (LLMs) in the semantic classification of free-text magnetic resonance imaging (MRI) reports.
MATERIALS AND METHODS: A corpus of 1385 reports from a single neuroradiology center was used to assess four LLMs (GPT-OSS-20B, Magistral-Small-2509, Gemma-4-31B, Qwen3.5-35B-A3B) of comparable scale (∼20-35B parameters). The brain MRI reports were labeled as Pathological or Non-Pathological, with the Pathological ones further classified into five diagnostic categories (Neoformation, Vascular, Inflammation/Infection, Neurodegenerative, Other). The LLMs were evaluated using zero- and few-shot prompting with majority-vote aggregation. Quantized Low-Rank Adaptation (QLoRA) fine-tuning was then applied to the best-performing model across multiple configurations. Performance was assessed using Macro F1, Micro F1, Precision, Recall, and Exact Match Accuracy.
RESULTS: The binary classification task achieved a Macro F1 score of 0.84 on the best-performing Qwen3.5-35B-A3B model, while multilabel classification reached 0.81. Majority voting (MV) outperformed individual-set predictions. QLoRA fine-tuning yielded a Macro F1 score of 0.83 on the multilabel task; the improvement over the 0-shot baseline was statistically supported for Macro F1, Micro F1, Macro Precision, and Exact Match, whose bootstrap 95% confidence intervals excluded zero, but not for Macro Recall. Fine-tuning and few-shot prompting strategies are complementary, with neither consistently outperforming the other. Error analysis showed that the models struggled with borderline annotation cases and often missed secondary labels.
CONCLUSION: Open-weight LLMs demonstrate strong capability for neuroradiology report classification without extensive task-specific training. Fine-tuning further improves performance, supporting integration into scalable semantic radiology data pipelines. However, the present evidence comes from a single Italian center without external validation, and the fine-tuning gain was estimated on a small held-out test set, for which the Macro F1 improvement, though statistically significant, has a confidence interval with a lower bound close to zero. In this light, the approach is best positioned as an internal indexing and semantic-retrieval tool, supporting faster report retrieval, structured cohort identification, internal audit, and scalable phenotyping for neuroradiology research and data warehousing, as well as efficient active-learning strategies, with broader generalizability requiring additional external multicentre validation before clinical deployment.
MATERIALS AND METHODS: A corpus of 1385 reports from a single neuroradiology center was used to assess four LLMs (GPT-OSS-20B, Magistral-Small-2509, Gemma-4-31B, Qwen3.5-35B-A3B) of comparable scale (∼20-35B parameters). The brain MRI reports were labeled as Pathological or Non-Pathological, with the Pathological ones further classified into five diagnostic categories (Neoformation, Vascular, Inflammation/Infection, Neurodegenerative, Other). The LLMs were evaluated using zero- and few-shot prompting with majority-vote aggregation. Quantized Low-Rank Adaptation (QLoRA) fine-tuning was then applied to the best-performing model across multiple configurations. Performance was assessed using Macro F1, Micro F1, Precision, Recall, and Exact Match Accuracy.
RESULTS: The binary classification task achieved a Macro F1 score of 0.84 on the best-performing Qwen3.5-35B-A3B model, while multilabel classification reached 0.81. Majority voting (MV) outperformed individual-set predictions. QLoRA fine-tuning yielded a Macro F1 score of 0.83 on the multilabel task; the improvement over the 0-shot baseline was statistically supported for Macro F1, Micro F1, Macro Precision, and Exact Match, whose bootstrap 95% confidence intervals excluded zero, but not for Macro Recall. Fine-tuning and few-shot prompting strategies are complementary, with neither consistently outperforming the other. Error analysis showed that the models struggled with borderline annotation cases and often missed secondary labels.
CONCLUSION: Open-weight LLMs demonstrate strong capability for neuroradiology report classification without extensive task-specific training. Fine-tuning further improves performance, supporting integration into scalable semantic radiology data pipelines. However, the present evidence comes from a single Italian center without external validation, and the fine-tuning gain was estimated on a small held-out test set, for which the Macro F1 improvement, though statistically significant, has a confidence interval with a lower bound close to zero. In this light, the approach is best positioned as an internal indexing and semantic-retrieval tool, supporting faster report retrieval, structured cohort identification, internal audit, and scalable phenotyping for neuroradiology research and data warehousing, as well as efficient active-learning strategies, with broader generalizability requiring additional external multicentre validation before clinical deployment.