Training Time of Prediff
yyyujintang opened this issue · 2 comments
yyyujintang commented
Could you provide the actual training time cost of the three models? According to the appendix, VAE, LDM, and KCN need 200, 1000, and 200 training epochs, respectively.
I tried on 3090 Ti and the experiment time costs are as follows.
4*3090 | Epochs | 1gpu | 2gpu | 4gpu | 4gpu Estimation (Day) |
---|---|---|---|---|---|
VAE-338MB | 200 | 2h | 1h 52min | 1h20min | 11.11 |
LDM-548MB | 1000 | 2h | 1h16min | 53min | 36.81 |
KCN-35MB | 200 | 30min | 15min | 8min | 1.11 |
gaozhihan commented
Thank you for your interest in our work and your question. It takes around 12 days for training the VAE on 4 NVIDIA A10G GPUs. It takes around 32 days for training the latent Earthformer-UNet using the same GPU configs.
yyyujintang commented
Thank you for your reply. It is very helpful.