Journal of the American College of Radiology : JACR · 2024
Multi-Institutional Evaluation and Training of Breast Density Classification AI Algorithm Using ACR Connect and AI-LAB.
СтатьяМногоцентровое исследованиеПоддержка NIH: внешняя программаПоддержка исследования: вне правительства СШАНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Brink Laura, Romero Ricardo Amaya, Coombs Laura, Tilkin Mike, Mazaheri Sina, Gichoya Judy, Zaiman Zachary, Trivedi Hari, Medina Adam, Bizzo Bernardo C, Chang Ken, Kalpathy-Cramer Jayashree, Kalra Mannudeep K, Astuto Bruno, Ramirez Carolina, Majumdar Sharmila, Lee Amie Y, Lee Christoph I, Cross Nathan M, Chen Po-Hao, Ciancibello Michael, Chiunda Allan, Nachand Douglas, Shah Chintan, Wald Christoph
Аннотация
OBJECTIVE: To demonstrate and test the capabilities of the ACR Connect and AI-LAB software platform by implementing multi-institutional artificial intelligence (AI) training and validation for breast density classification.
METHODS: In this proof-of-concept study, six US-based hospitals installed Connect and AI-LAB. A breast density algorithm was trained and tested on retrospective mammograms. We recorded time to receive institutional review board approval, to install software locally, and to complete the testing and training. We calculated the performance of the breast density algorithm at each participating hospital and compared it to the performance of a holdout multi-institutional clinical trial testing dataset and a retrospective multi-institutional dataset. We calculated the performance of the locally fine-tuned models on the holdout test datasets.
RESULTS: The median time to receive institutional review board approval was 66 days, and the median time to successfully install Connect and AI-LAB locally was 157 days. The median time to complete breast density algorithm testing and training was 216 days. The breast density algorithm performed worse at each hospital than on the holdout test dataset, suggesting poor generalizability of the base model. The fine-tuned models had mixed performance locally and performed poorly on the test dataset.
DISCUSSION: In this study, we demonstrate the successful installation and implementation of Connect and AI-LAB software platforms at six facilities using a breast density algorithm. Our results suggest poor generalizability of an algorithm trained on a single dataset and algorithms fine-tuned at individual institutions, emphasizing the hypothetical importance of multi-institutional testing and training.
METHODS: In this proof-of-concept study, six US-based hospitals installed Connect and AI-LAB. A breast density algorithm was trained and tested on retrospective mammograms. We recorded time to receive institutional review board approval, to install software locally, and to complete the testing and training. We calculated the performance of the breast density algorithm at each participating hospital and compared it to the performance of a holdout multi-institutional clinical trial testing dataset and a retrospective multi-institutional dataset. We calculated the performance of the locally fine-tuned models on the holdout test datasets.
RESULTS: The median time to receive institutional review board approval was 66 days, and the median time to successfully install Connect and AI-LAB locally was 157 days. The median time to complete breast density algorithm testing and training was 216 days. The breast density algorithm performed worse at each hospital than on the holdout test dataset, suggesting poor generalizability of the base model. The fine-tuned models had mixed performance locally and performed poorly on the test dataset.
DISCUSSION: In this study, we demonstrate the successful installation and implementation of Connect and AI-LAB software platforms at six facilities using a breast density algorithm. Our results suggest poor generalizability of an algorithm trained on a single dataset and algorithms fine-tuned at individual institutions, emphasizing the hypothetical importance of multi-institutional testing and training.