Reduce memory consumption to prevent errors due to container OOM
minimaxir opened this issue · 2 comments
minimaxir commented
Containers seem to go OOM after ~10 generations, despite garbage collection. Loading the model takes up ~1.5GB so hitting the ceiling is not surprising, but there should be a way to control the leaks.
minimaxir commented
The current implementation (reload model after 8 generations) appears to avoid OOMs.
deepbluesea commented
this will reduce the memory consumption by a lot tensor2tensor.utils.adafactor.adafactoroptimizer