Implementation of OpenAI paper "An Empirical Model of Large-Batch Training" for Fastai V2.
The code is based on the batch size finder implementation for Fastai V1 by DanyWind (repo V1 / blog / discussion).
This implementation differs on:
- It implements exactly the original article and not an aproximation (by default).
- Fixes a couple of bugs in noise and scale values. However, they didn't affect on Simple Noise Scale value.
However, you could use the DanyWind aproximation by settting simulate_multi_gpus to False. DanyWind aproximation is faster but numerically more inestable and finds a Simple Noise Scale smaller than the original Simple Noise Scale.
It's tested with fastai 2.1. It should work with fastai>=2.0
TODO:
- Port description improvements from fastai2 PR.