This is an unofficial PyTorch implementation of the SEED (ICLR-2021):
We implement SEED based on the official code of MoCo.
Teacher model is MoCo-v2 (top-1: 67.6) pretrained on ImageNet-1k with ResNet-50. We distill it to ResNet-18. Results show that our code is credible.
| SEED | Top-1 acc | Top-5 acc |
|---|---|---|
| Official results (hidden_dim=512) | 57.60 | 81.80 |
| **Ours** (hidden_dim=512) | 58.03 | 82.44 |
| **Ours** (hidden_dim=2048) | 60.32 | 83.50 |
| **Ours** (symmetry, hidden_dim=2048) | 61.27 | 84.06 |
Hidden dimension (hidden_dim) can be modified by
self.encoder_q.fc = nn.Sequential(nn.Linear(dim_smlp, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, dim))
Install PyTorch and ImageNet dataset following the official PyTorch ImageNet training code.
This repo aims to be minimal modifications on MoCo. Running by:
sh train.sh
More student architectures.
@inproceedings{fang2021seed,
author = {Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu},
title = {SEED: Self-supervised Distillation for Visual Representation},
booktitle = {ICLR},
year = {2021},
}