/TAPADL

Code of "Robustifying Token Attention for Vision Transformers"

Primary LanguagePythonMIT LicenseMIT

Robustifying Token Attention for Vision Transformers

Yong Guo, David Stutz, and Bernt Schiele. ICCV 2023.

Paper | Slides | Poster

This repository contains the official Pytorch implementation and the pretrained models of Robustifying Token Attention for Vision Transformers.

Catalog

  • Pre-trained models for image classification
  • Pre-trained models for semantic segmentation
  • Evaluation and Training Code

Dependencies

Our code is built based on pytorch and timm library. Please check the detailed dependencies in requirements.txt.

Dataset Preparation

  • Image Classfication: ImageNet and related robustness benchmarks

Please download the clean ImageNet dataset. We evaluate the models on varisous robustness benchmarks, including ImageNet-C, ImageNet-A, ImageNet-P, and ImageNet-R.

  • Semantic Segmentaton: Cityscapes and related robustness benchmarks

Please download the clean Cityscapes dataset. We evaluate the models on varisous robustness benchmarks, including Cityscapes-C and ACDC (test set).

Training and Evaluation (using TAP and ADL)

  • Image Classification:

    Please see how to train/evaluate FAN and RVT models in TAPADL_FAN and TAPADL_RVT, respectively.

  • Semantic Segmentation:

    Please see how to train/evaluate our segmentation model in TAPADL_FAN/segmentation.

Acknowledgement

This repository is built using the timm library, RVT, and FAN repositories.

Citation

If you find this repository helpful, please consider citing:

@inproceedings{guo2023robustifying,
title={Robustifying token attention for vision transformers},
author={Guo, Yong and Stutz, David and Schiele, Bernt},
booktitle={Proceedings of the IEEE International Conference on Computer Vision (ICCV)}},
year={2023}
}