SegVol: Universal and Interactive Volumetric Medical Image Segmentation

🚀News: The information and links of original datasets have been collected in the Supplementary Materials of paper preprinted in arXiv.

The SegVol is a universal and interactive model for volumetric medical image segmentation. SegVol accepts point, box and text prompt while output volumetric segmentation. By training on 90k unlabeled Computed Tomography (CT) volumes and 6k labeled CTs, this foundation model supports the segmentation of over 200 anatomical categories.

We have released SegVol's inference code, training code, model params and ViT pre-training params (pre-training is performed over 2,000 epochs on 96k CTs).

Key words: 3D medical SAM, volumetric image segmentation

Quickstart

Requirements

The pytorch v1.11.0 (or higher version) is needed first. Following install key requirements using commands:

pip install 'monai[all]==0.9.0'
pip install einops==0.6.1
pip install transformers==4.18.0
pip install matplotlib

Guideline for training and inference

How to infer a demo case.

How to train SegVol.

How to use our pre-trained ViT as your model encoder.

Datasets involved

Dataset	Link
3D-IRCADB	https://www.kaggle.com/datasets/nguyenhoainam27/3dircadb
AbdomenCT-1k	https://github.com/JunMa11/AbdomenCT-1K
AMOS22	https://amos22.grand-challenge.org/
BTCV	https://www.synapse.org/\#!Synapse:syn3193805/wiki/217752
CHAOS	https://chaos.grand-challenge.org/
CT-ORG	https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=61080890
FLARE22	https://flare22.grand-challenge.org/
HaN-Seg	https://han-seg2023.grand-challenge.org/
KiPA22	https://kipa22.grand-challenge.org/
KiTS19	https://kits19.grand-challenge.org/
KiTS23	https://kits-challenge.org/kits23/
LUNA16	https://luna16.grand-challenge.org/Data/
MSD-Colon	http://medicaldecathlon.com/
MSD-HepaticVessel	http://medicaldecathlon.com/
MSD-Liver	http://medicaldecathlon.com/
MSD-lung	http://medicaldecathlon.com/
MSD-pancreas	http://medicaldecathlon.com/
MSD-spleen	http://medicaldecathlon.com/
Pancreas-CT	https://wiki.cancerimagingarchive.net/display/public/pancreas-ct
QUBIQ	https://qubiq.grand-challenge.org/
SLIVER07	https://sliver07.grand-challenge.org/
TotalSegmentator	https://github.com/wasserth/TotalSegmentator
ULS23	https://uls23.grand-challenge.org/
VerSe19	https://osf.io/nqjyw/
VerSe20	https://osf.io/t98fz/
WORD	https://paperswithcode.com/dataset/word

Web Tool of SegVol 📽

segvol.mp4

Internal Validation Performance🏆

External Validation Performance🏆

We performed an external validation experiment using a novel annotated dataset from the ULS23 Challenge (750 + 744 + 124 cases about lesions) and the validation dataset from Amos22 (120 cases about organs). SegVol showed strong segmentation abilities compared to other medical SAM methods in accurately segmenting lesions and 15 important organs.

Visualization🔍

Dataset (We will release our dataset shortly)

Internal Validation

External Validation

News🚀

(2024.01.03) A radar map about zero-shot experiment has been reported. 🏆

(2023.12.25) Our web tool supports download results now! You can use it as an online tool. 🔥🔥🔥

(2023.12.15) The training code has been uploaded!

(2023.12.04) A web tool of SegVol is here! Just enjoy it! 🔥🔥🔥

(2023.11.28) Our model and demo case have been open-source at huggingface/BAAI/SegVol. 🤗🤗

(2023.11.28) The usage of pre-trained ViT has been uploaded.

(2023.11.24) You can download weight files of SegVol and ViT(CTs pre-train) from huggingface/BAAI/SegVol or Google Drive. 🔥🔥🔥

(2023.11.23) The brief introduction and instruction have been uploaded.

(2023.11.23) The inference demo code has been uploaded.

(2023.11.22) The first edition of our paper has been uploaded to arXiv. 📃

Citation

If you find this repository helpful, please consider citing:

@article{du2023segvol,
  title={SegVol: Universal and Interactive Volumetric Medical Image Segmentation},
  author={Du, Yuxin and Bai, Fan and Huang, Tiejun and Zhao, Bo},
  journal={arXiv preprint arXiv:2311.13385},
  year={2023}
}