[Trans-YOLOv5: a YOLOv5-based prior transformer network model for automated detection of abnormal cells or clumps in cervical cytology images].

The development of various models for automated images screening has significantly enhanced the efficiency and accuracy of cervical cytology image analysis. Single-stage target detection models are capable of fast detection of abnormalities in cervical cytology, but an accurate diagnosis of abnormal cells not only relies on identification of a single cell itself, but also involves the comparison with the surrounding cells. Herein we present the Trans-YOLOv5 model, an automated abnormal cell detection model based on the YOLOv5 model incorporating the global-local attention mechanism to allow efficient multiclassification detection of abnormal cells in cervical cytology images. The experimental results using a large cervical cytology image dataset demonstrated the efficiency and accuracy of this model in comparison with the state-of-the-art methods, with a mAP reaching 65.9% and an AR reaching 53.3%, showing a great potential of this model in automated cervical cancer screening based on cervical cytology images.