Integration of Transformer-Based Architecture and Large Language Models for Optical Coherence Tomography Data Analysis to Improve the Accuracy of Differential Diagnosis of Retinal Diseases.

UNLABELLED: The aim of the present study was to improve the differential diagnosis accuracy of retinal diseases combining a transformer model to classify the biomarkers on OCT images and the large language model DeepSeek-V3.

MATERIALS AND

METHODS

Two datasets were collected and annotated: a training set (3288 central retinal OCT images annotated for 8 biomarkers) and a validation set (50 clinical cases from octcases.com). For biomarker classification, we compared ResNet, DenseNet, EfficientNet, and Vision Transformer (ViT-Tiny-Patch16-224) architectures. The ViT-Tiny model demonstrated the highest performance, its F1 macro - 0.84±0.03. An integration algorithm was developed to combine predicted biomarker labels with clinical history data via the DeepSeek-V3 LLM API.

RESULTS

Combining OCT biomarkers and the medical history significantly improved the diagnostic accuracy: Top-1 Accuracy - 78%, Top-3 Accuracy - 94%, MRR - 84%, representing 10-44% improvement over using either data type alone.

CONCLUSION

The suggested approach enabled to automate the detection of biomarkers on OCT images and enhance the differential diagnosis accuracy of eye diseases, reducing the image interpretation time and supporting clinical decision-making.