Code for Synthesizing Knowledge-enhanced Features for Real-world Zero-shot Food Detection

Environment requirements

- mmdetection we recommend using Docker 2.0.

- The code implementation of our experiments mainly based on PyTorch 1.1.0 and Python 3.6.

The following scripts are for different steps in the pipeline on PASCAL VOC dataset, please see the respective files for more arguments. Before running the scripts, please set the datasets and backbone paths in the config files. Weights of ResNet101 trained excluding overlapping unseen classes from ImageNet.

1. Train object detector on seen classes

   cd mmdetection
   ./tools/dist_train.sh configs/pascal_voc/faster_rcnn_r101_fpn_1x_voc0712.py

2. Extract features

   # extract seen classes features to train synthesizer and unseen class features for cross validation
   
   cd mmdetection
   
   # extract training features for seen classes
   python tools/zero_shot_utils.py configs/pascal_voc/faster_rcnn_r101_fpn_1x_voc0712.py --classes seen --load_from ./work_dir/voc0712/epoch_4.pth --save_dir ../../data/voc --data_split train
   
   # extract training features for unseen classes
   python tools/zero_shot_utils.py configs/pascal_voc/faster_rcnn_r101_fpn_1x_voc0712.py --classes unseen --load_from ./work_dir/voc0712/epoch_4.pth --save_dir ../../data/voc --data_split test

3. Train synthesizer

  # modify the paths to extracted features, labels and model checkpoints.
  ./script/train_gcn_synthesizer.sh

4. Test

   cd mmdetection
   
   ## C setting
   ./tools/dist_test.sh configs/pascal_voc/faster_rcnn_r101_fpn_1x_voc0712.py /work_dir/voc0712/epoch_4.pth --dataset voc --out voc_results.pkl --zsd --syn_weights ../checkpoints/VOC/classifier_best.pth
   
   ## G setting
   ./tools/dist_test.sh configs/pascal_voc/faster_rcnn_r101_fpn_1x_voc0712.py /work_dir/voc0712/epoch_4.pth --dataset voc --out voc_results.pkl --gzsd --syn_weights ../checkpoints/VOC/classifier_best.pth

LanceZPF/KEFS