Final report can be found here
S2C-MMT can be found here
Before running our code, the environment preperation and data installation steps provided in Scan2Cap and 3DETR repositories must be completed.
After the installation step, make sure to change the data-paths in lib/config.py. The lines needed to change are marked with # TODO
Important: The given model and the code base is tested in PyTorch 1.8.0. We do not guarantee compatibility with later versions.
To load the pretrained 3DETR models, download 3DETR-m and 3DETR files into pretrained/ folder. This step can be skipped if the models trained by us are going to be loaded.
The following script can be used to reproduce our results for 3DETR-S2C:
python scripts/eval.py --folder 3DETRS2C --eval_caption --min_iou [0.25 or 0.50] --eval_detection --enc_type masked --no_height --enc_dropout 0 --use_topdown --num_proposals 256 --num_locals 10 --use_relation --num_graph_steps 2
python scripts/visualize.py --scene_id <scene_id> --folder 3detrs2c --no_height --use_topdown --use_relation --use_orientation --num_graph_steps 2 --num_proposals 256 --num_locals 10 --enc_type masked --enc_dropout 0
python scritps/train.py --lr 1e-3 --wd=1e-4 [--enc_type masked] --enc_dropout 0.3 --use_normal --use_topdown --num_proposals 256 --num_locals 10 --batch_size 8 --epoch 50 --use_relation --num_graph_steps 2
python scripts/train_pretrained3detr.py [--unfreeze_3detr] [--enc_type masked] --lr 1e-4 --wd=1e-5 --no_height --enc_dropout 0.3 --use_topdown --num_proposals 256 --num_locals 10 --batch_size 8 --epoch 50 --use_relation --num_graph_steps 2
python scripts/eval.py --folder <folder containing the model> --eval_caption --min_iou [0.25 or 0.50] [--eval_detection] [--enc_type masked] --no_height --enc_dropout 0 --use_topdown --num_proposals 256 --num_locals 10 --use_relation --num_graph_steps 2
Scan2Cap is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.
The majority of 3DETR is licensed under the Apache 2.0 license.