luogen1996/RepAdapter

Official implementation of "Towards Efficient Visual Adaption via Structural Re-parameterization".

RepAdapter

Official implementation of "Towards Efficient Visual Adaption via Structural Re-parameterization". Repadapter is a parameter-efficient and computationally friendly adapter for giant vision models, which can be seamlessly integrated into most vision models via structural re-parameterization. Compared to Full Tuning, RepAdapter saves up to 25% training time, 20% GPU memory, and 94.6% storage cost of ViT-B/16 on VTAB-1k.

Updates

• (2023/02/16) Release our RepAdapter project.

Data Preparation

We provide two ways for preparing VTAB-1k:

• Download the source datasets, please refer to NOAH.
• We provide the prepared datasets, which can be download from google drive.

After that, the file structure should look like:

$ROOT/data
|-- cifar
|-- caltech101
......
|-- diabetic_retinopathy

• Download the pretrained ViT-B/16 to ./ViT-B_16.npz

Training and Evaluation

1. Search the hyper-parameter s for RepAdapter (optional)

bash search_repblock.sh

2. Train RepAdapter

bash train_repblock.sh

3. Test RepAdapter

python test.py --method repblock --dataset <dataset-name> 

Usage Example

The following is a simple example of using RepAdapter to load and train a model

1. Import repadapter.py from module

from repadapter import set_repadapter, save_repadapter,load_repadapter

2. Insert RepAdapter layers into all linear layers in the model

set_repadapter(model=model)

If you need to train only specific linear layers, you can modify set_repadapter to use regular expressions to match specific names.

import re
import torch.nn as nn
def set_repadapter(model, pattern):
    # Compile regular expression patterns
    regex = re.compile(pattern)
    for name, module in model.named_modules():
        # Check if the module is a linear layer and if the name matches a regular expression
        if isinstance(module, nn.Linear) and regex.match(name):

3. Set the requires_grad attribute of the model parameters.

• Set the requires_grad attribute of the model parameters as needed to determine which parameters require training.

trainable = []
for n, p in model.named_parameters():
    if any([x in n for x in ['repadapter']]):
        trainable.append(p)
        p.requires_grad = True
    else:
        p.requires_grad = False

4. Save the checkpoint of RepAdapter

• After training is completed, generally only the repadapter parameters of the model are saved. This can save a significant amount of disk space, which is one of the advantages of using RepAdapter.

import os
save_repadapter(os.path.join(output_dir,"final.pt"), model=model)

5. Load the checkpoint of RepAdapter

• If you need to load a model that has been saved after training, the model needs to execute set_repadapter before loading.

load_repadapter(load_path, model=model)

6. Reparameterize the model

• merge_repadapter is used after model training to simplify the model structure, reducing the model size and inference time.
• merge_repadapter takes the model and the save path of the repadapter as inputs and performs reparameterization.

merge_repadapter(model,load_path=None,has_loaded=False)

Citation

If this repository is helpful for your research, or you want to refer the provided results in your paper, consider cite:

@article{luo2023towards,
  title={Towards Efficient Visual Adaption via Structural Re-parameterization},
  author={Luo, Gen and Huang, Minglang and Zhou, Yiyi  and Sun, Xiaoshuai and Jiang, Guangnan and Wang, Zhiyu and Ji, Rongrong},
  journal={arXiv preprint arXiv:2302.08106},
  year={2023}
}