dahyun-kang/ifsl

Multi shot implementation does not exit

Ehteshamciitwah opened this issue · 5 comments

Ehteshamciitwah commented

Hello,

Thank you for sharing your work. As you stated that you are using 1 shot for the training and than use multiple forward for K shot during evaluation. While passing shot more than 1 in evaluation mode raise an error if self.args.shot > 1: raise NotImplementedError
Can you please share your multiple-shot evaluation code? https://github.com/dahyun-kang/ifsl/blob/eb2cf9604ff5d55e3a158d25ebc27fcb32ada067/fs-cs/model/ifsl.py#L106

dahyun-kang commented

Hi Ehtesham

If you just want to check the quantitative performance, you can exclude --vis when running the evaluation.
Thank you and have a good day! 😃

Best,
Dahyun

Ehteshamciitwah commented

Thank you for your quick response.
You mean visualization for multishot is not implemented but we can get quantitative performance with multi-shot.

dahyun-kang commented

Yes, exactly.

Ehteshamciitwah commented

Hello,
I think your visualization code also works with multi-shots, but the only last image from the shot will be considered as support image. Due to your model test code structure, the batch inputs change from multishot to one shot automatically because of may be clone() https://github.com/dahyun-kang/ifsl/blob/eb2cf9604ff5d55e3a158d25ebc27fcb32ada067/fs-cs/model/asnet.py#L68
Furthermore, for multi-shot, performance is not always guaranteed to be increased. In some images performance decrease with multi-shots.

Ehteshamciitwah commented

Hello, can you please elaborate?
To extend the model to the multi-shot. There are mostly two techniques. One is trained on 1 shot and performs k shot forward for multi-shot and averages their results. Like Asnet.
Second, perform k-shot training directly.
Which method shows more performance improvement ?
and How we can Train ASnet directly on multishot?