/ptorch_examples

Pytorch implementations of deep learning models

Primary LanguageJupyter Notebook

Deep Learning examples in Pytorch

SimCLR

Simple Pytorch implementation in src/simclr/simclr.py.
An example notebook is provided where we look at a simplistic contrastive learning task, distinguishing points on the unit circle (available here and in Google Colab Open In Colab).
I wrote this to illustrate my recent blog post about SimCLR.
The original paper is available here.

ResNet re-implementations

The code in src/models/resnet.py is a re-implementation of the original ResNet paper.

Next steps are implementing ideas from:

  • The "Bag of tricks" paper
  • The "revisiting ResNets" paper

Results on CIFAR-10's test set:

Setup Accuracy
Baseline 0.915
Baseline+ResNet simple tweaks 0.915

Baseline:

  • 200 epochs
  • learning rate 0.1
  • cosine decay
  • linear warmup of 5 epochs
  • batch size 128
  • weight decay 1e-4

ResNet simple tweaks:

  • better downsampling, instead of stride 2 1x1 convolutions
  • 3 convolutions 3x3 instead of a single 7x7 convolution in the stem

TODOs:

  • More training epochs with / without MixUp
  • Label smoothing
  • Reduce weight decay