/Contrastive_Representation_Distillation

Primary LanguagePython

Contrastive_Representation_Distillation

Implementation of CRD (Contrastive Representation Distillation)

Quick Start

Teacher training

python main.py --option "teacher"

Student training without distillation

python main.py --option "student"

Student training with distillation

python main.py --option "distill" --teacher_model [teacher_model]

Test

python main.py --option [option] --test --prev_model [prev_model]

Result

Model Teacher Student (No Distill) Studnet (Distill)
Beta - - 0.7 0.8 0.9 1.0 1.1
Accuracy 77.08% 76% 75.92% 75.57% 76.08% 76.27% 75.74%

Reference

  1. Contrastive Representation Distillation (https://arxiv.org/abs/1910.10699)
  2. CIFAR-100 Dataset (https://www.cs.toronto.edu/~kriz/cifar.html)
  3. ResNet implementation (https://github.com/weiaicunzai/pytorch-cifar100/blob/master/models/resnet.py)