Neuro-oncology · 2026
Comprehensive performance testing and external validation of an AI algorithm to detect and segment brain metastases.
СтатьяВалидационное исследованиеНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Kotecha Rupesh, Akdemir Eyub Y, Din Naseem Ud, Yarlagadda Sreenija, Reyes Mauricio, Nitti Francesco, Gutierrez Alonso N, Wieczorek D Jay M, Lee Yongsook C, Tolakanahalli Ranjini, Bander Evan D, McDermott Michael W, Ahluwalia Manmeet S, Mehta Minesh P
Аннотация
BACKGROUND: Artificial intelligence (AI)-based models have shown initial promise in imaging brain metastasis; however, many lack validation against advanced imaging-informed datasets, precluding external validity and limiting widespread adoption. To overcome these limitations, we performed comprehensive performance testing against reference standard metrics and externally validated an AI algorithm.
METHODS: As part of its FDA-clearance process, performance testing of a previously developed U-Net-based AI model was conducted on a multi-institutional cohort with reference standard established via consensus review by 3 neuroradiologists. External validation was performed on patients imaged with dual sequences (augmented) as well as an open-access dataset (UCSF-BMSR). Evaluation metrics included sensitivity, false positive (FP) rate, positive predictive value (PPV), dice similarity coefficient (DSC), 95% Hausdorff distance, normalized surface distance, and qualitative physician assessment.
RESULTS: In the FDA performance testing cohort, the AI algorithm achieved a sensitivity of 90.0% (95% CI: 87.0%-94.0%), DSC of 0.86 (95% CI: 0.83-0.89), and average FP rate of 0.57 lesions. In the augmented and open-access external validation cohorts, a sensitivity of 81.4% (95% CI: 73.7%-89.1%) and 85.2% (95% CI: 83.0%-87.4%) with an average number of 0.22 and 1.19 FP lesions and DSCs of 0.70 (95% CI: 0.66-0.73) and 0.78 (95% CI: 0.77-0.78) were calculated, respectively. In the augmented external validation cohort, 38.5% of contours were rated as requiring major revisions.
CONCLUSION: This AI algorithm demonstrated promising performance via 3 unique datasets. However, given the notable rate of contour revisions, these findings support its clinical role not as an autonomous system but as a human-in-the-loop tool requiring physician oversight.
METHODS: As part of its FDA-clearance process, performance testing of a previously developed U-Net-based AI model was conducted on a multi-institutional cohort with reference standard established via consensus review by 3 neuroradiologists. External validation was performed on patients imaged with dual sequences (augmented) as well as an open-access dataset (UCSF-BMSR). Evaluation metrics included sensitivity, false positive (FP) rate, positive predictive value (PPV), dice similarity coefficient (DSC), 95% Hausdorff distance, normalized surface distance, and qualitative physician assessment.
RESULTS: In the FDA performance testing cohort, the AI algorithm achieved a sensitivity of 90.0% (95% CI: 87.0%-94.0%), DSC of 0.86 (95% CI: 0.83-0.89), and average FP rate of 0.57 lesions. In the augmented and open-access external validation cohorts, a sensitivity of 81.4% (95% CI: 73.7%-89.1%) and 85.2% (95% CI: 83.0%-87.4%) with an average number of 0.22 and 1.19 FP lesions and DSCs of 0.70 (95% CI: 0.66-0.73) and 0.78 (95% CI: 0.77-0.78) were calculated, respectively. In the augmented external validation cohort, 38.5% of contours were rated as requiring major revisions.
CONCLUSION: This AI algorithm demonstrated promising performance via 3 unique datasets. However, given the notable rate of contour revisions, these findings support its clinical role not as an autonomous system but as a human-in-the-loop tool requiring physician oversight.