A heatmap-based deep learning framework for multi-modal registration of VIS, NIR, and thermal images in dermatological imaging.

INTRODUCTION

Multi-modal image registration leverages complementary information from diverse imaging sources to achieve precise spatial alignment. However, aligning visible (VIS), near-infrared (NIR), and thermal (TH) modalities remains challenging due to appearance differences and limited annotated datasets.

METHODS

This study proposes a ResU-Net-inspired framework combining heatmap prediction and homography estimation to enable joint NIR-TH registration via shared feature representations. A proprietary dataset of 155 VIS-NIR-TH skin lesion triplets was created from 62 patients, resulting in 465 images in total. Diagnoses were confirmed by dermatologists or histopathology. Lesions were distributed across the face, trunk, and limbs to evaluate registration robustness under spatial variability.

RESULTS AND DISCUSSION: Experiments showed that NIR-VIS registration consistently outperformed TH-VIS registration, reflecting NIR's richer structural content and higher spectral similarity with VIS. Despite limitations related to dataset size and acquisition variability, the framework demonstrated the feasibility of VIS-NIR-TH triplet registration and provides the first documented dataset of its kind for multi-modal skin lesion imaging.