CASIA-IVA-Lab/DPT

How to use DPT in DETR

lingblessing opened this issue · 5 comments

lingblessing commented

Does the author add the DPT module to the DETR part? A little anxious, thank you very much

volgachen commented

We simply replace the PVT backbone with our DPT model, without any other modification in DETR decoder & encoder layers.

Please see the configuration here
https://github.com/CASIA-IVA-Lab/DPT/blob/main/detection/configs/detr_dpt_s_8x2_50ep_coco.py

lingblessing commented

It is to replace the ResNet50 in DETR with PVT, and the Transformer is still connected later, right?
And DPT replaces a module in PVT, right?

volgachen commented

Yeah.
For how to replace ResNet50 with PVT, please refer to the paper of PVT.

lingblessing commented

Thank you very much for your patient answer. In DETR, the backbone output dimension is 256 and the number of channels is 2048. How do you set the dimension and number of channels in DPT?

volgachen commented

The dimension of backbone output is 512, while the transformer dimension is set to be 256.
There is a layer to handle the dimension transformation.

For our configuration please refer detr_r50_8x2_50ep_coco_baseline.py

For detailed implementation please refer mmdet/models/dense_heads/transformer_head.py in mmdetection (v2.8.0).