Look into stability of weight initialization.
Opened this issue · 0 comments
dleebrown commented
Under more extensive testing, it looks like the weight initialization in v0.1.0, which follows the He+ (2015) weight initialization scheme, is unstable under certain training conditions, namely low absolute variance in the input data (e.g. training on a <1000 K temperature range).
- Is this a bug? Is my weight initialization incorrect vs He+?
- If not a bug, redo the weight initialization to make the training process more stable.
For now, rolling back to the previous weight initialization scheme, which is just based on the incoming dimensionality to each layer.