Reproducibility

[mdraw]

• All PyTorch RNGs should be seeded with a configurable seed in example scripts.
• NumPy RNGs in worker processes should use deterministic seeds that can be controlled from training scripts. Worker seeds need to be based on the PyTorch seed, but we have to manually make sure they are different for each forked process.

NumPy RNG reproducibility has high priority because the order of training samples for network training can matter, especially if we want to do clean hyperparameter optimization.
We probably don't want to aim for "maximum reproducibility" (torch.backends.cudnn.deterministic etc.). If someone really wants this, they can set it themselves in custom training scripts.

References:

• https://pytorch.org/docs/master/notes/randomness.html
• https://pytorch.org/docs/master/data.html#torch.utils.data.DataLoader ("NOTE")
• numpy/numpy#9650