IsoBN
Code for AAAI 2021 paper IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization.
Requirements
- PyTorch (tested on 1.7.0)
- Transformers (tested on 3.4.0)
- wandb
- tqdm
- datasets
Training and Evaluation
Train the BERT-base model on GLUE with the following command:
>> python train.py --task_name TASK
The training loss and evaluation results on the dev set are synced to the wandb dashboard.