CSCI567

file structure

To evalua all the method, make sure evaluation data and sam model in this structure:

segment-anything
├── sam_vit_h_4b8939.pth
└── datasets/people_poses
    ├── train_images/*.jpg
    ├── train_segementations/*.png
    ├── val_images/*.jpg
    ├── val_segementations/*.png
    ├── train_id.txt
    ├── val_id.txt
    └── val_id_short.txt

dataset readme
val_id_short.txt include the 20 first image from val set for visualization and comparation of all method
SAM model download by

wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

The evaluation file and linear probe training file structure:

segment-anything
└── notebooks
    ├── eval
    │   ├── xxx_result
    │   │   ├── miou.csv
    │   │   └── pixacc.csv
    │   ├── *.py
    │   ├── *_colab.ipynb
    │   └── *_local_machine.ipynb
    ├── train
    │   ├── *.py
    │   ├── gen_embedding.ipynb
    │   └── gen_embedding.ipynb
    └── vis
        ├── *.py
        ├── *.ipynb
        ├── *.png
        └── *.npy

in eval folder:
- *_colab.ipynb: small evaluation test on colab
- *_local_machine.ipynb: small evaluation test on local mechine
- *.py: evaluation for whole val data set on local mechine
in train folder:
- gen_embedding.ipynb: generate image embeding by SAM for speeding up the trainning processe
- train_embedding*.ipynb: try different version of linear probe structure setup and test program
- train_embedding.py: train final version linear probe method based on image embeddings
in vis folder:
- compare_vis.ipynb: compare predicted masks generated by different model for different class in val_id_short.txt
- *_local_machine.ipynb: small evaluation test on local mechine
- *.py: evaluation for whole val data set on local mechine

preparation

python>=3.8, as well as pytorch>=1.7 and torchvision>=0.8.

Add people_poses data set into datasets dataset readme and link
Add Sam model to this repo root folder

train linear probe model

generate image embeding by gen_embedding.ipynb
use ipynb or py file in the train folder to train the linear probe model

evaluation

use *_local_machine.ipynb to setup different model enviroment, and try on small data amount of data and get some visualization result
use *.py file to generate the csv result for each model (the interactive method based on image embedding data)

cd notebooks/eval
python <model_name>.py

reulst will show up in notebooks/eval/*result folder

model names

The list of possible models that can be model_name are (these are the models you can evaluate):

CLIP_+_SAM (CLIP + SAM model)
clipseg_sam (CLIPSeg + SAM model)
glip_sam (GLIP + SAM model)
groundingdino_sam (Grounding DINO + SAM model)
interactive_embedding (1-Point Interactive Segmentation models (random & center) (oracle/not oracle))