Gastrointestinal endoscopy · 2024
Creating a standardized tool for the evaluation and comparison of artificial intelligence-based computer-aided detection programs in colonoscopy: a modified Delphi approach.
СтатьяСравнительное исследованиеПоддержка исследования: вне правительства СШАНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Gadi Sanjay R V, Mori Yuichi, Misawa Masashi, East James E, Hassan Cesare, Repici Alessandro, Byrne Michael F, von Renteln Daniel, Hewett David G, Wang Pu, Saito Yutaka, Matsubayashi Carolina Ogawa, Ahmad Omer F, Sharma Prateek, Gross Seth A, Sengupta Neil, Mansour Nabil, Cherubini Andrea, Dinh Nhan Ngo, Xiao Xiao, Mountney Peter, González-Bueno Puyal Juana, Little Greg, LaRocco Shawn, Conjeti Sailesh, Seibt Hannes, Zur Dror, Shimada Hitoshi, Berzin Tyler M, Glissen Brown Jeremy R
Аннотация
BACKGROUND AND AIMS: Multiple computer-aided detection (CADe) software programs have now achieved regulatory approval in the United States, Europe, and Asia and are being used in routine clinical practice to support colorectal cancer screening. There is uncertainty regarding how different CADe algorithms may perform. No objective methodology exists for comparing different algorithms. We aimed to identify priority scoring metrics for CADe evaluation and comparison.
METHODS: A modified Delphi approach was used. Twenty-five global leaders in CADe in colonoscopy, including endoscopists, researchers, and industry representatives, participated in an online survey over the course of 8 months. Participants generated 121 scoring criteria, 54 of which were deemed within the study scope and distributed for review and asynchronous e-mail-based open comment. Participants then scored criteria in order of priority on a 5-point Likert scale during ranking round 1. The top 11 highest priority criteria were re-distributed, with another opportunity for open comment, followed by a final round of priority scoring to identify the final 6 criteria.
RESULTS: Mean priority scores for the 54 criteria ranged from 2.25 to 4.38 after the first ranking round. The top 11 criteria after round 1 of ranking yielded mean priority scores ranging from 3.04 to 4.16. The final 6 highest priority criteria, including a tie for first-place ranking, were (1, tied) sensitivity (average, 4.16) and (1, tied) separate and independent validation of the CADe algorithm (average, 4.16); (3) adenoma detection rate (average, 4.08); (4) false-positive rate (average, 4.00); (5) latency (average, 3.84); and (6) adenoma miss rate (average, 3.68).
CONCLUSIONS: This is the first reported international consensus statement of priority scoring metrics for CADe in colonoscopy. These scoring criteria should inform CADe software development and refinement. Future research should validate these metrics on a benchmark video dataset to develop a validated scoring instrument.
METHODS: A modified Delphi approach was used. Twenty-five global leaders in CADe in colonoscopy, including endoscopists, researchers, and industry representatives, participated in an online survey over the course of 8 months. Participants generated 121 scoring criteria, 54 of which were deemed within the study scope and distributed for review and asynchronous e-mail-based open comment. Participants then scored criteria in order of priority on a 5-point Likert scale during ranking round 1. The top 11 highest priority criteria were re-distributed, with another opportunity for open comment, followed by a final round of priority scoring to identify the final 6 criteria.
RESULTS: Mean priority scores for the 54 criteria ranged from 2.25 to 4.38 after the first ranking round. The top 11 criteria after round 1 of ranking yielded mean priority scores ranging from 3.04 to 4.16. The final 6 highest priority criteria, including a tie for first-place ranking, were (1, tied) sensitivity (average, 4.16) and (1, tied) separate and independent validation of the CADe algorithm (average, 4.16); (3) adenoma detection rate (average, 4.08); (4) false-positive rate (average, 4.00); (5) latency (average, 3.84); and (6) adenoma miss rate (average, 3.68).
CONCLUSIONS: This is the first reported international consensus statement of priority scoring metrics for CADe in colonoscopy. These scoring criteria should inform CADe software development and refinement. Future research should validate these metrics on a benchmark video dataset to develop a validated scoring instrument.