Skeletal radiology · 2026
Multi-reader, multi-model benchmark of large language models for modified Outerbridge cartilage grading from knee MRI reports.
СтатьяНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Zhao Anna H, Roux Elizabeth L, Wieschhoff Ged, Bahouth Sara, Smith Stacy E, Wu Jim S, Mandell Jacob C
Аннотация
OBJECTIVE: To describe a reproducible framework for bulk large language model (LLM)-based extraction of structured cartilage-lesion data from knee MRI reports and to benchmark seven LLM configurations against multiple radiologists using the modified Outerbridge classification.
MATERIALS AND METHODS: In this IRB-approved retrospective study, 100 non-contrast knee MRI reports (January 2019 to January 2025) were randomly selected from 66,479 eligible examinations and independently graded by five readers (four fellowship-trained musculoskeletal radiologists with 6-21 years of post-fellowship experience and one fourth-year resident) and seven LLM configurations, comprising six Azure OpenAI deployments (GPT-4.1, GPT-5.1-mini, GPT-5.3, GPT-5.4, GPT-5.4-mini, GPT-5.4-nano) and one locally hosted open-weight model (Qwen2.5-32B-Instruct), across a fixed 20-surface anatomic taxonomy. Pairwise Cohen quadratic-weighted κ was computed at compartment and surface levels. Bootstrap 95% confidence intervals (CIs) and p-values on the paired difference between each rater's mean κ and the human-human reference were derived by case-level cluster resampling (B = 1000).
RESULTS: Mean human-human κ was 0.94 (95% CI 0.93, 0.95) at the compartment level and 0.80 (95% CI 0.78, 0.83) at the surface level. Three Azure deployments-GPT-5.4 (surface κ = 0.83), GPT-5.3 (κ = 0.80), and GPT-5.1-mini (κ = 0.77)-had 95% CIs on the difference from the human reference that included zero. Four models had CIs entirely below zero, indicating agreement significantly below the human reference: GPT-4.1, GPT-5.4-mini, Qwen2.5-32B, and GPT-5.4-nano (all p ≤ .02).
CONCLUSION: Three flagship OpenAI deployments achieved cartilage grading agreement indistinguishable from human inter-rater variability, while cost-optimized and open-weight variants performed measurably below.
MATERIALS AND METHODS: In this IRB-approved retrospective study, 100 non-contrast knee MRI reports (January 2019 to January 2025) were randomly selected from 66,479 eligible examinations and independently graded by five readers (four fellowship-trained musculoskeletal radiologists with 6-21 years of post-fellowship experience and one fourth-year resident) and seven LLM configurations, comprising six Azure OpenAI deployments (GPT-4.1, GPT-5.1-mini, GPT-5.3, GPT-5.4, GPT-5.4-mini, GPT-5.4-nano) and one locally hosted open-weight model (Qwen2.5-32B-Instruct), across a fixed 20-surface anatomic taxonomy. Pairwise Cohen quadratic-weighted κ was computed at compartment and surface levels. Bootstrap 95% confidence intervals (CIs) and p-values on the paired difference between each rater's mean κ and the human-human reference were derived by case-level cluster resampling (B = 1000).
RESULTS: Mean human-human κ was 0.94 (95% CI 0.93, 0.95) at the compartment level and 0.80 (95% CI 0.78, 0.83) at the surface level. Three Azure deployments-GPT-5.4 (surface κ = 0.83), GPT-5.3 (κ = 0.80), and GPT-5.1-mini (κ = 0.77)-had 95% CIs on the difference from the human reference that included zero. Four models had CIs entirely below zero, indicating agreement significantly below the human reference: GPT-4.1, GPT-5.4-mini, Qwen2.5-32B, and GPT-5.4-nano (all p ≤ .02).
CONCLUSION: Three flagship OpenAI deployments achieved cartilage grading agreement indistinguishable from human inter-rater variability, while cost-optimized and open-weight variants performed measurably below.