How did you infer in zero shot setting, since we don't have bboxes in zero shot and model is built to accept bboxes as function parameter?

[KNITPhoenix]

How did you infer in zero shot setting, since we don't have bboxes in zero shot and model is built to accept bboxes as function parameter?

[djukicn]

Hi @KNITPhoenix, in the zero-shot setting, we replace the appearance and shape queries by learnable objectness queries and use them as initial prototypes in the OPE module (see Section 3.1.1 in the paper). The model does accept bboxes as parameter, but when zero_shot is set to True, it will be ignored (so you can set it to None or anything else).

[KNITPhoenix]

Thanks for the reply. I understand that and getting the following error when passing only image to the zero shot model:

File "evaluate_sccpg.py", line 94, in
evaluate(args)
File "/home/spandey8/anaconda3/envs/loca/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "evaluate_sccpg.py", line 67, in evaluate
out, _ = model(img)
File "/home/spandey8/anaconda3/envs/loca/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/home/spandey8/anaconda3/envs/loca/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1040, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "/home/spandey8/anaconda3/envs/loca/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1000, in _run_ddp_forward
return module_to_run(*inputs[0], **kwargs[0])
File "/home/spandey8/anaconda3/envs/loca/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
TypeError: forward() missing 1 required positional argument: 'bboxes'

Since its a zero shot scenario, so I am not passing any annotations for the image and the annotation file being passed is an empty json file. Also, below is the complete shell file for evaluation that I am using:

torchrun --nproc_per_node=1 evaluate_sccpg.py
--model_name loca_zero_shot
--data_path /home/spandey8/Object_counting/sccpg_original_data/annotations/
--model_path /home/spandey8/Object_counting/loca/pretrained_models
--backbone resnet50
--swav_backbone
--reduction 8
--image_size 512
--num_enc_layers 3
--num_ope_iterative_steps 3
--emb_dim 256
--num_heads 8
--epochs 200
--lr 1e-4
--backbone_lr 0
--lr_drop 300
--weight_decay 1e-4
--batch_size 16
--dropout 0.1
--num_workers 6
--max_grad_norm 0.1
--aux_weight 0.3
--tiling_p 0.5
--pre_norm
--zero_shot

Please guide me in how to proceed for a zero shot evaluation.

[djukicn]

Replacing the line that errors out out, _ = model(img) with out, _ = model(img, None) should do the trick.

[KNITPhoenix]

Thanks a lot.