Hardware requirements?

Question

Hardware requirements?

daqingliu opened this issue 5 years ago · 6 comments

Hi, thanks for your awesome paper and codes.
Recently I'm preparing to follow your work and I'm wondering what hardware it requires and how long it will cost while training?
Thanks!

Answer 1 · 2019-09-03T15:45:20.000Z

Emmmm, I have not tested the minimal requirement of computational resource but I believe that GTX 1080 would be enough.

Answer 2 · 2019-09-03T15:48:36.000Z

How about the training time?

Answer 3 · 2019-09-03T15:50:40.000Z

Speaker and Agent finishes in around 6 hours. Agent + Speaker + EnvDrop converges slowly, which takes around 1 day to converge.

Answer 4 · 2019-09-03T15:51:42.000Z

Thank you very much :)

Answer 5 · 2019-10-26T12:21:12.000Z

I train Speaker and Agent on two RTX 2080Ti in parallel and it takes 1 day for 80000 iterations. The GPU usage is about 30~40 since sampling data in RL takes time. Half of my CPUs are idle and half of memory are free during training.

I come to ask if you know how to promote training efficiency and what is your suggestion?

Answer 6 · 2019-10-26T19:24:03.000Z

Yep, I am aware that the GPUs are not fully utilized. But I am sorry that I do not know the answer.

Previously, the speed bottleneck is CPU-GPU data transfer (it's quite slow on pageable memory) and I optimize it a lot. So I am not sure whether CPU-based sampling would speed it up.