This repository provides an implementation of the CVPR 2021 oral paper Open-Vocabulary Object Detection Using Captions. The code is built on top of Facebook's maskrcnn-benchmark. We have also partially used some code from Facebook's ViLbert and HuggingFace's transformers. We appreciate the work of everyone involved in those invaluable projects.
We provide a simple demo that creates a side-by-side video of a regular Faster R-CNN vs. our open-vocabulary detector. To run, just open any of the notebooks inside the demo folder.
Check INSTALL.md for installation instructions. For the demo to create the video output, it might be necessary to build OpenCV from source instead of installing using pip.
For the following examples to work, you need to download the COCO dataset.
We recommend to symlink the path to the coco dataset to datasets/. Refer to path_catalog.py for the names of the required files. After setting up the dataset, run:
python -m torch.distributed.launch --nproc_per_node=8 tools/train_net.py --config-file configs/mmss_v07.yaml --skip-test OUTPUT_DIR ~/runs/vltrain/121For the zero-shot experiment to work, you need to first create a new annotation json using this notebook. Then run:
python -m torch.distributed.launch --nproc_per_node=8 tools/train_net.py --config-file configs/zeroshot_v06.yaml OUTPUT_DIR ~/runs/maskrcnn/130You can evaluate using a similar command as above, by running tools/test_net.py and providing the right checkpoint path to MODEL.WEIGHT
Our best model is available for download here, and has been trained using this config.
The pretrained model before fine-tuning on object detection can be found here, and has been trained using this config.
Note: If the links are broken, please contact me directly.
We did not test all the functionality of maskrcnn_benchmark under the zero-shot settings, such as instance segmentation, or feature pyramid network. Anything besides the provided config files may not work.
Created and maintained by Alireza Zareian.