how to accelerate the training process?
fikry102 opened this issue · 2 comments
fikry102 commented
python ./main.py --config ./configs/rectified_flow/cifar10_rf_gaussian_ddpmpp.py --eval_folder eval --mode train --workdir ./logs/1_rectified_flow
It seems that the training process need 60w iterations.
It seems that the memory usage of each gpu is not very high during the training process. (4.3G for 24G RTX3090)
Is there any way to increase the memory usage and therefore accelerate the training process?
forever208 commented
just increase the batch size and learning rate at the same time
forever208 commented
@fikry102 Hi, how long does it take to train on CIFAR-10? for example, with batch=128, how many steps and how many hours does it cost on a RTX 3090?