[Reimplementation] How are you associating ROI features to GT bounding boxes?
peterstratton opened this issue · 6 comments
Prerequisite
- I have searched Issues and Discussions but cannot get the expected help.
- I have read the FAQ documentation but cannot get the expected help.
- The bug has not been fixed in the latest version (master) or latest version (3.x).
💬 Describe the reimplementation questions
I've been reading through and rerunning the code and I haven't been able to figure out how you associate ROI features to their corresponding GT bboxes in the current batch - as specified by algorithm 1 in the paper.
I noticed on lines 434 and 435 of feature_imitation_roi_head.py, we have:
cur_gt_cat_id = sampling_results[i].pos_gt_labels[hq_gt_ind]
cur_gt_roi_feat = bbox_feats[i][hq_gt_ind, :, :, :].clone()
My understanding is that hq_gt_ind corresponds to the index of the ground truths that has enough predictions to compute the necessary IQ score. Thus, it will have a value in [0, ... , num ground truths].
However, bbox_feats[i] has shape (256, 256, 7, 7) and sampling_results[i].pos_gt_labels has a shape larger than the number of ground truths.
If I'm understanding right, why does indexing into sampling_results[i].pos_gt_labels and bbox_feats[i] with hq_gt_ind make sense?
Environment
/opt/conda/lib/python3.7/site-packages/mmcv/init.py:21: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
'On January 1, 2023, MMCV will release v2.0.0, in which it will remove '
sys.platform: linux
Python: 3.7.13 (default, Mar 29 2022, 02:18:16) [GCC 7.5.0]
CUDA available: True
GPU 0,1: NVIDIA RTX A6000
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 11.3, V11.3.109
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.12.1
PyTorch compiling details: PyTorch built with:
- GCC 9.3
- C++ Version: 201402
- Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 11.3
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
- CuDNN 8.3.2 (built against CUDA 11.5)
- Magma 2.5.2
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,
TorchVision: 0.13.1
OpenCV: 4.8.1
MMCV: 1.7.1
MMCV Compiler: GCC 7.5
MMCV CUDA Compiler: 11.3
MMDetection: 2.26.0+unknown
Expected results
No response
Additional information
No response
When I train the project ，something wrong happens. In datasets , the config file named sodad.py
has key "ori_ann_file" , and it's val defualt as "rawData/Annotations/val_wo_ignore.json". I do not know its meaning?
For a single image, we have the ground truth with the number of num_gts, then the shape of bbox_feats[i] is (num_pro, 256, 7, 7) and the term num_pro is set to 512 by default, which indicates the number of proposals., and the shape of sampling_results[i].pos_gt_labels is (num_pos_pro, ), and num_pos_pro ≥ num_gts.
Moreover, the first num_gts regional features in bbox_feats correspond to the num_gts ground truths in the current image.
We use hq_gt_ind to index the high-quality instances, and if you take a closer look at it, you will find the maximum number of hq_gt_ind is less than or equal to num_gts, i.e., max(hq_gt_ind) ≤ num_gts - 1. The above process actually corresponds to the line 6 of Alg. 1 in our paper.
@jiinhui Filter ignore annotations in val.json then you will get val_wo_ignore.json
Hey Shaun, thanks for the response! That all makes sense and I get that those lines correspond to line 6: saving the roi features corresponding to gt bboxes that have high IQ associated with them.
More specifically: how do you know that bbox_feats[i][hq_gt_ind, :, :, :] is the roi feature that corresponds to the gt of hq_gt_ind? Why do they match up?
I think I understand what is supposed to be happening conceptually, but I'm not seeing how the code is doing what the paper is stating.
Sry, I could't clearly understand what you mean. When you get hq_gt_ind, you can easily obtain the desired roi features by the index operation in Python.
One thing for sure is that the first num_gts regional features in bbox_feats correspond to the num_gts ground truths in the current image, this is the default setting in MMDetection. Maybe you can take a deeper look at bbox_feats in the standard_roi_head.py.
Ok sounds good, I'll check out bbox_feats in standard_roi_head.py. I didn't know that the first num_gts were automatically associated to the num_gts ground truths. Thank you!