Multimodal deep learning for predicting WHO/ISUP grading in renal tumors on CT using a self-attention-based model: variable Vision Transformer (vViT).

PURPOSE: To investigate the performance of the variable vision transformer (vViT) model in predicting the World Health Organization/International Society of Urological Pathology (WHO/ISUP) grade in renal tumors using multimodal data, including clinical information, radiomic features, and computed tomography (CT) images, and to identify the most dominant factor among these inputs.

MATERIALS AND

METHODS

The vViT model was trained and validated on 111 patients (1398 images) and 15 patients (224 images), respectively, to classify renal tumors into low (WHO/ISUP grades 1-2) and high (grades 3-4) grades. The trained model was tested on 30 patients (228 images). The model inputs included class token, patient demographics (age, sex, body mass index), comorbidities (peripheral vascular disease, diabetes mellitus, chronic kidney disease), habits (smoking, alcohol use), radiomics, and CT images. Permutation feature importance was used to evaluate the contribution of each input sector. The model's performance was compared with other models; ViT, ConvNeXt, and ResNeXt using the DeLong test.

RESULTS

vViT achieved an accuracy of 0.811 and an area under the receiver operating characteristic curve (AUC-ROC) of 0.856 for the test dataset and outperformed ViT and ResNeXt (p < 0.05). In both predictions, the radiomic features were identified as the most dominant predictor of WHO/ISUP grade (p < 0.05).

CONCLUSION

The performance of vViT had no significant difference compared with other models in predicting the WHO/ISUP pathological grade in renal tumors using multimodal data. These results highlight the potential of integrating radiomic analysis with clinical data in grading of renal tumors.