where is ckpt_path = "focalnet_base_lrf.pth" and attention or focus scores

Hello, Jianwei:

I am trying to use focalnet to get some attention/focus scores for my images, I see it perform well here in the modulation map of the below images, and this is exactly what I need, I want to calculate each pixel's attention/focus score, I see in your image, the yellow is with highest attention scores (not sure attention score is the right name, basically I want to see which pixels get more attention from people's eyes).

And I have two questions:

I try to run the code in visualizaiton.ipynb but cannot find "focalnet_base_lrf.pth", where to download it please?
I wonder can you give a hint, from which code part, I can get attention scores which shows in the modulation map.

Thank you very much for your help in advance!

Best wishes,
Wen

I think I get the answer to my first question: "focalnet_base_lrf.pth" is actually one model of ImageNet-1K Pretrained FocalNet-B.
Not sure the second one...

Hi, @WenY2020 ,

thanks for your interest in our work. Regarding your two questions:

You can download the 'focalnet_base_lrf.pth' from this link: https://projects4jw.blob.core.windows.net/focalnet/release/classification/focalnet_base_lrf.pth.
actually, you can find all the downloadable ckpts from here: https://github.com/microsoft/FocalNet#imagenet-1k-pretrained

This checkpoint is hierarchical which is slightly worse for visualization. I recommend you can also try to use the isotropic focalnets.

For visualize the modulation map, you can simply refer to this line of code:

FocalNet/classification/focalnet.py

Line 96 in 6de0705

x_out = q*self.modulator

It stores the modulator for each focal modulation block so that you can visualize the magnitude of the modulation maps for all focal modulation layers.

thanks,
Jianwei

And if you just want to visualize your images, you can directly use this hugging face demo!

https://huggingface.co/spaces/jw2yang/focalnet-modulators

thank you very much, Jianwei, those are very helpful,
I wll look into here: x_out = q*self.modulator,
Great, I need not only the visualization, mainly I need the attention numbers/scores, will need to use it as a feature to feed other models.

Awesome, let me know if you have any further questions.

thanks you, will do:).