Memory Leak
Opened this issue · 2 comments
Hi!
We trained the model from scratch on Gibson dataset with the setting that num_scenes = 72, but we found that once the program is running, the avaliable memory of our machine is going lower and lower, and exhausted after about 2 hours.
We found it leaks over time and stop leaking after we terminate the program.
The curve of avaliable memory is shown as follows.
We also found that if we reduce the number of scenes, the speed of memory leaking will also decrease.
It will be very helpful if someone can suggest the possible solution.
Hi!
Too long FIFOMemory length may be the cause of Memory Leaking.
Default value of slam_memory_size is set to 500000, the traning data will be appended to the FIFOMemory untill the FIFO length reaches the maximum of 500000, which continuously takes up a lot of memory.
We set slam_memory_size to 720 hence the queue stop growing and stop consuming the memory.
The large default value of slam_memory_size is confusing. Does it matter a lot to the performance of the model?
Yes, the memory increase is due to the FIFO memory. In our experiments, we found the memory helps in improving sample efficiency but not the final performance, if you reduce the memory size, the model might require more frames to reach the same performance.