Leakage-Aware Visit-Level Benchmarking Reveals Representational Overlap in Deep Learning for Bacterial Versus Fungal Keratitis Classification.

PURPOSE: The purpose of this study was to benchmark deep learning for bacterial versus fungal keratitis classification from slit-lamp photographs under a leakage-aware, visit-level framework, and to characterize embedding-space geometry as a potential performance ceiling.

METHODS

This retrospective single-center study included white-light slit-lamp photographs from 101 patients with culture-confirmed bacterial or fungal keratitis (258 visits; 658 images), using strict patient-disjoint five-fold cross-validation with the clinical visit as the primary prediction unit. Image-level classifiers, multiple instance learning (MIL), and DINOv2-based retrieval were compared under three preprocessing strategies: full-frame, corneal region of interest (ROI), and lesion-centered MedSAM crops. Embedding geometry was analyzed using Uniform Manifold Approximation and Projection (UMAP). Pairwise visit-level model comparisons were performed using patient-cluster paired bootstrap on aligned out-of-fold (OOF) visit-level predictions, and probability calibration was assessed using expected calibration error (ECE) and Brier score.

RESULTS

At the image level, the highest area under the receiver operating characteristic curve (AUROC) was 0.691. At the visit level, convolutional neural network (CNN)-based MIL models achieved the highest AUROC (0.677), followed by retrieval-based approaches (0.661). However, MIL models showed larger generalization gaps, whereas DINOv2 lesion-crop retrieval remained stable (-0.014). Lesion-centered preprocessing was the dominant performance determinant within the retrieval paradigm. Embedding analysis revealed extensive class overlap, with 90.9% of fungal visits within the bacterial convex hull.

CONCLUSIONS

Under leakage-aware, visit-level evaluation, all paradigms converged to modest discrimination, consistent with representational overlap within this single-center cohort. CNN-based MIL achieved higher AUROC but greater overfitting; retrieval-based models provided more stable generalization and superior probability calibration. Whether this ceiling is task-intrinsic or site-specific requires multicenter validation.

TRANSLATIONAL RELEVANCE: Leakage-aware visit-level benchmarking supports realistic evaluation and calibrated decision support.