/rnn_darts_fastai

Implement Differentiable Architecture Search (DARTS) for RNN with fastai

Primary LanguageJupyter Notebook

DARTS for RNN with fastai

Language model on Penn Treebank using Differentiable Architecture Search (DARTS) and fastai library.
Blog post.
Based on DARTS: Differentiable Architecture Search by Hanxiao Liu, Karen Simonyan, Yiming Yang.
Check out the original implementation.

Requirements

fastai 1.0.52.dev0 (latest as of 10th April 2019), PyTorch 1.0.

Instructions

  1. Run databunch_nb.ipynb to create databunch
  2. Run train_search_nb.ipynb to search for genotype. ~5 hours on 1 v100 gpu for 1 run.
    RNN search is sensitive to initialization so there should be several runs with different seed
  3. Train that genotype from scratch on train_nb.ipynb. ~1.5 days for 1600 epochs.
  4. Test a model using test_nb.ipynb

Pretrained model

Pretrained model of DARTS_V1 genotype after 600 epochs darts_V1.pth.
Place the file at data/models and run test_nb.ipynb. Loss ~4.22, 68.0 perplexity.
Caveat: I haven't been able to get ~58.0 test perplexity like the original implementation.

fastai dev version installation

git clone https://github.com/fastai/fastai
cd fastai
tools/run-after-git-clone
pip install -e ".[dev]"
git pull