Determination of an optimal quality control method for Pap test analysis using digital cytology and artificial intelligence.

INTRODUCTION

The aim of this study was to determine an optimal method for performing quality control on Pap tests that adhere to CLIA '88 mandates that are applicable to employing an artificial intelligence (AI)-based system in routine practice.

MATERIALS AND

METHODS

Four hundred ninety-seven archival ThinPrep Pap slides were retrieved and scanned with Genius Digital Diagnostics System. A cytologist and 3 cytopathologists (CP) performed initial interpretations (first read). Subsequently, all slides were rescanned and the new output was re-reviewed by another cytologist and the same 3 CP's after a washout period of 14 months (second read). The result of serial reads of each pathologist was compared with the original ThinPrep Interpretation (OTPI) and with each other. Interobserver concordance was calculated for all reads.

RESULTS

Out of 497 cases 32.6%, 17.9%, 17.9%, 6.8%, 24.1%, and 0.6% were interpreted as negative for intraepithelial lesion or malignancy, atypical squamous cells of undetermined significance, low-grade squamous intraepithelial, atypical squamous cells, cannot exclude a high-grade squamous intraepithelial lesion, high-grade squamous intraepithelial lesion/squamous cell carcinoma, and atypical glandular cell/adenocarcinoma in situ/adenocarcinoma per OTPI. Kendall's coefficient for concordance amongst the 3 CP's for serial reads were 0.924 and 0.892, respectively. Concordance between 4 diagnostic results (3 CP results and OTPI) for first and second reads was 0.902 and 0.865, respectively. Three hundred seven (61.8%), 310 (62.4%), and 328 (66%) cases were consistent when serial reads were compared for pathologist A, B, and C, respectively. Kappa values between reads ranged from 0.51 to 0.559 amongst the pathologists.

CONCLUSIONS

Regular quality control checks that involve rescanning a subset of randomly selected slides, reanalyzing them with AI and multiple interpretations from CP's can be adopted for quality assurance for AI-assisted Pap test screening in the future after regulatory guidelines are updated.