Amending AI Software Accuracy for Diabetic Retinopathy Detection Using Conditional Probability and the Appropriate Reference Standard.

AIM: To evaluate and correct the reported sensitivity and specificity values of DART (Diagnóstico Automatizado de Retinografías Telemáticas), an automated Artificial Intelligence based (AI-based) screening tool used for diabetic retinopathy (DR) detection in the Chilean public healthcare system, by employing the appropriate gold standard and conditional probability.

METHODS

Data were obtained from the clinical validation of DART. DR detection capabilities were assessed for three methods 1) Fundoscopy, 2) Retinography and 3) AI- DART. To estimate the true sensitivity and specificity of DART, conditional probability was applied using three hypothetical sensitivities level for method 2: A) Optimistic (90%), B) Moderate (80%), and C) Pessimistic (70%). Based on these scenarios, corrected sensitivity and specificity values for DART were calculated, along with false negative/positive rates (%FN/%FP), and predictive values (NPV/PPV).

RESULTS

In all scenarios, corrected sensitivity and specificity values for DART were significantly lower than those reported in the original validation study. Compared to method 3 (AI-based), method 2 (retinography by and ophthalmologist) consistently demonstrated superior performance across all metrics, including FN%, FP%, NPV and PPV values.

CONCLUSION

While the integration of new AI-based technologies like DART in healthcare offer promise for enhancing patient care, their implementation must be preceded by validation using the correct gold standard. Reliable clinical decision-making depends on trustworthy diagnostic parameters.