About Train time
diamond0910 opened this issue · 1 comments
Nice work!
I'm running this repo on 8 NVIDIA GeForce 3090 Cards.
Here are some logs:
[000000] d: 1.2211; g: 2.6081; real: -0.2209; fake: -0.6971; r1_img: 0.0001; r1_seg: 0.0000; path: 0.0256; mean path: 0.0016; mask: 0.5293; time: 14.90[000100] d: 0.1674; g: 2.8209; real: 2.8159; fake: -2.3380; r1_img: 0.0001; r1_seg: 0.0006; path: 0.0007; mean path: 0.0137; mask: 0.0038; time: 1.77
[000200] d: 0.1239; g: 3.3147; real: 3.9278; fake: -3.2984; r1_img: 0.0000; r1_seg: 0.0002; path: 0.0027; mean path: 0.0235; mask: 0.0057; time: 1.76[000300] d: 0.0870; g: 4.0814; real: 11.0209; fake: -4.2459; r1_img: 0.0000; r1_seg: 0.0001; path: 0.0001; mean path: 0.0329; mask: 0.0040; time: 1.75
It's seem that 1.77 * 20000 / 60 / 60 / 24 = 4.1 days are need to train. The paper says 'For the 512×512 model, our model takes about two and a half day to train 150,000 steps with a batch size of 32 on 8 32GB Nvidia Tesla V100 GPUs, where the best model is then selected.'
What time for each iter do V100 GPU cost？Can you show the log duaring training? Thank you!
I don't have the old training log anymore. The average time for each step on this repo is also about 1.8s.