jsikyoon/dreamer-torch

replay data memory usage?

tominku opened this issue · 5 comments

tominku commented

Hi, thank you for the good code base.
I just wonder if a normal PC can embrace all the replay data in the memory when the agent step goes over 1 million. If I have about 16 GB memory, then can this agent be trained until the end?
It seems like the replay data size keep increasing as the training proceeds. Do you have any idea that the agent can be trained in a small-sized memory?
Thanks!

jsikyoon commented

Hi tominku,

Thank you for your interesting on my codes!

As I know, Dreamer stores the reply buffer on the disk not memory, but did you experience the memory leak on training? Memory leak means increasement of the memory usage while training.

Bests, Jaesik.

tominku commented

I'm looking around the code before the full run. I noticed that

  1. the code saves an episode at disk everytime an epispde is finished,

  2. But it also maintains the episode in the memory thus every new episode is added to the memory replay buffer.

  3. In training, the agent samples data from the memory, not from the disk. It assumes that all the available episodes data are stored in the memory. Please check the code.

jsikyoon commented

Hi tominku,

Yes, you are right. I assumed they load reply buffer from the disk when it requires because they stored the episodes on disk, but it is not.

One thing we can choose is to modify the codes to load the reply buffer from disk not storing it in memory.

Or a hybrid way could be the reply buffer is stored in memory to the pre-defined limitation and beyond that, they can load it from disk. -> It is harder to implement than naively loading reply buffer from disk I think.

Bests, Jaesik.

tominku commented

Thanks! That might be a good solution.

jsikyoon commented

If you have no comments for this issue, I will close :)