training gpu cost

Question

training gpu cost

zhw-zhang opened this issue a year ago · 3 comments

Hi, thank you for your open-source code. How much GPU consumption is required during training? Is it necessary to add additional deepspeed or checkpoint to save training memory consumption?

Answer 1 · 2023-12-20T08:50:12.000Z

This depends on your specific training setup, with my current configuration (512*320, 16 frames, with all parameters of unet trainable), it will use approximately 63GB of VRAM (I haven't enabled any optimizations, such as deepspeed and so on)

Answer 2 · 2023-12-20T10:03:40.000Z

Thank you for your reply. So your batchsize is 1? At 512*320 resolution.

Answer 3 · 2023-12-21T04:07:15.000Z

Yes, batchsize=1