TransResNet: Integrating the Strengths of ViTs and CNNs for High Resolution Medical Image Segmentation via Feature Grafting (BMVC 2022) (Will updated soon)
Muhammad Hamza Sharif, Dmitry Demidov, Asif Hanif, Mohammad Yaqub, and Min Xu
Abstract: High-resolution images are preferable in medical imaging domain as they significantly improve the diagnostic capability of the underlying method. In particular, high resolution helps substantially in improving automatic image segmentation. However, most of the existing deep learning-based techniques for medical image segmentation are optimized for input images having small spatial dimensions and perform poorly on high-resolution images. To address this shortcoming, we propose a parallel-in-branch architecture called TransResNet, which incorporates Transformer and CNN in a parallel manner to extract features from multi-resolution images independently. In TransResNet, we introduce Cross Grafting Module (CGM), which generates the grafted features, enriched in both global semantic and low-level spatial details, by combining the feature maps from Transformer and CNN branches through fusion and self-attention mechanism. Moreover, we use these grafted features in the decoding process, increasing the information flow for better prediction of the segmentation mask. Extensive experiments on ten datasets demonstrate that TransResNet achieves either state-of-the-art or competitive results on several segmentation tasks, including skin lesion, retinal vessel, and polyp segmentation.
Cross Grafting Module
See INSTALL.md for the installation of dependencies required to run TransResNet.
To test the pre-trained TransResNet models of Skin-Segmentation, Polyp-Segmentation, and Retinal-Vessel-Segmentation on your images you can either use the following command as:
python
Example usage to perform Skin-Segmentation on a directory of images:
python
Example usage to perform Retinal-Vessel-Segmentation on a directory of images:
python
Example usage to perform Polyp-Segmentation on a directory of images:
python
Training and Testing instructions for Skin, Polyp, and Retinal-Vessel segmentation are provided in their respective directories. Here is a summary table containing hyperlinks for easy navigation:
| Task | Training Instructions | Testing Instructions |
|---|---|---|
| Skin Segmentation | Link | Link |
| Polyp Segmentation | Link | Link |
| Retinal Vessel Segmentation | Link | Link |
Experiments are performed for different medical segmentation tasks including, polyp, skin, and retinal-vessel.
Skin Segmentation (click to expand)
Retinal Vessel Segmentation (click to expand)
Polyp Segmentation (click to expand)
Qualitative Results (click to expand)
If you use TransResNet, please consider citing:
@inproceedings{Sharif2022TransResNet,
title={TransResNet: Integrating the Strengths of ViTs and CNNs for High Resolution Medical Image Segmentation via Feature Grafting},
author={Muhammad Hamza Sharif and Dmitry Demidov and Asif Hanif and Mohammad Yaqub and Min Xu},
booktitle={BMVC},
year={2022}
}
Should you have any question, please contact sharifmhamza@gmail.com