CellViT++: Energy-efficient and adaptive cell segmentation and classification using foundation models.

BACKGROUND AND OBJECTIVE: Deep learning-based cell segmentation and classification methods in digital pathology are critical for diagnostics but are hampered by models that require extensive annotated datasets, are computationally expensive, and lack adaptability to new cell types. This creates a significant bottleneck in research and clinical workflows. This study introduces CellViT++, a data-efficient and lightweight framework for generalized cell segmentation that allows for rapid adaptation to novel cell taxonomies with minimal data.

METHODS: CellViT++ leverages a Vision Transformer with a frozen pretrained foundation model for segmentation. It simultaneously extracts deep cell embeddings from the transformer tokens during the forward pass at no extra computational cost. To adapt to new cell types, only a lightweight classifier is trained on these embeddings, bypassing the need to retrain the segmentation model. We also demonstrate an automated workflow to generate training data from registered H&E and immunofluorescence (IF) slides. The framework was validated on seven public datasets.

RESULTS: The framework achieves remarkable zero-shot segmentation results and data efficiency. On the CoNSeP dataset for colon cancer, we achieved superior results with only 10% of the training data. On all other datasets, we outperformed competing methods or at least approached their performance, all in one model. The classifier approach, based on zero-shot segmentation models, drastically reduces computational costs, with training times of minutes versus hours for baseline models, decreasing CO2 emission by 96.93%. Models trained on automatically generated labels from IF-staining achieved performance comparable to (lymphocytes, ΔF1:-0.042) or even exceeding (plasma cells, ΔF1:+0.108) those trained on expert-annotated datasets.

CONCLUSIONS: CellViT++ provides a robust and efficient open-source framework that addresses key limitations in computational pathology by decoupling segmentation from classification. Its ability to adapt to new cell types with minimal data and its support for automated dataset generation from IF slides significantly reduce the reliance on time-consuming expert annotation. This work provides a foundational tool to accelerate research, enhance diagnostic workflows, and enable deeper cohort analysis. The code is available under https://github.com/TIO-IKIM/CellViT-plus-plus and as a PyPI package.