FeatUp: A Model-Agnostic Framework for Features at Any Resolution

ICLR 2024

Stephanie Fu*, Mark Hamilton*, Laura Brandt, Axel Feldman, Zhoutong Zhang, William T. Freeman *Equal Contribution.

TL;DR:FeatUp improves the spatial resolution of any model's features by 16-32x without changing their semantics.

teaser.2.mp4

Install
Using Pretrained Upsamplers
Fitting an Implicit Upsampler
Coming Soon
Citation
Contact

Install

Pip

For those just looking to quickly use the FeatUp APIs install via:

pip install git+https://github.com/mhamilton723/FeatUp

Local Development

To install FeatUp for local development and to get access to the sample images install using the following:

git clone https://github.com/mhamilton723/FeatUp.git
cd FeatUp
pip install -e .

Using Pretrained Upsamplers

To see examples of pretrained model usage please see our Collab notebook. We currently supply the following pretrained versions of FeatUp's JBU upsampler:

Model Name	Checkpoint	Torch Hub Repository	Torch Hub Name
DINO	Download	`mhamilton723/FeatUp`	`dino16`
DINO v2	Download	`mhamilton723/FeatUp`	`dinov2`
CLIP	Download	`mhamilton723/FeatUp`	`clip`
ViT	Download	`mhamilton723/FeatUp`	`vit`
ResNet50	Download	`mhamilton723/FeatUp`	`resnet50`

For example, to load the FeatUp JBU upsampler for the DINO backbone:

upsampler = torch.hub.load("mhamilton723/FeatUp", 'dino16')

Fitting an Implicit Upsampler to an Image

To train an implicit upsampler for a given image and backbone first clone the repository and install it for local development. Then run

cd featup
python train_implicit_upsampler.py

Parameters for this training operation can be found in the implicit_upsampler config file.

Coming Soon:

Training your own FeatUp joint bilateral upsampler
Simple API for Implicit FeatUp training
Pretrained JBU models without layer-norms

Citation

@inproceedings{
    fu2024featup,
    title={FeatUp: A Model-Agnostic Framework for Features at Any Resolution},
    author={Stephanie Fu and Mark Hamilton and Laura E. Brandt and Axel Feldmann and Zhoutong Zhang and William T. Freeman},
    booktitle={The Twelfth International Conference on Learning Representations},
    year={2024},
    url={https://openreview.net/forum?id=GkJiNn2QDF}
}

Contact

For feedback, questions, or press inquiries please contact Stephanie Fu and Mark Hamilton

devendra-dantal04/FeatUp