Comparing the Generalizability of Multiregional versus Locally Trained Deep Learning Models for Trachoma Detection.

OBJECTIVE

To test the advantage of geographically diverse, multiregional training of artificial intelligence models over single-region training for detection of trachomatous inflammation-follicular (TF) across test sets from different regions.

DESIGN

A comparative study evaluating the performance of trachoma detection models trained on single-region image datasets versus multiregional datasets.

SUBJECTS: A total of 71 206 everted eyelid photographs from 15 605 subjects aged from 0 to 9 years.

METHODS

Everted eyelid photographs from Ethiopia, Niger, and Peru were collected between 2014 and 2020. They were graded for the presence or absence of TF by certified experts. The resulting labels were used to train, validate, and test 3 single-region models and multiregional models for trachoma detection.

MAIN OUTCOME MEASURES: The F1-score, area under the receiver operating characteristic curve (AUROC), and predicted TF prevalence were used to measure performance of all models on each of the test sets from Ethiopia, Niger, and Peru. Heatmaps were generated to visualize the highest areas of activation.

RESULTS

Results are reported for the Ethiopia, Niger, and Peru test sets in that order. Except for Niger, single-region models performed best on their respective local test sets with F1 = 0.78; 0.24; 0.89 and AUROC = 0.94; 0.97; 0.98, and accurate prevalence predictions. However, they performed significantly worse on data from other regions. In contrast, the multiregional model accurately estimated trachoma prevalence in all test sets across all regions and matched the performance of the single-region models on their respective test sets (F1 = 0.85; 0.25; 0.89 and AUROC = 0.96; 0.79; 0.99). Heatmaps showed agreement with diagnostically relevant trachoma features.

CONCLUSIONS

Geographically diverse training data are essential to developing broadly generalizable deep learning models for trachoma detection. Using such models to guide mass distribution of macrolide antibiotics may improve the scalability and cost effectiveness of trachoma control campaigns.

FINANCIAL DISCLOSURES: Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.