Tranining problems

[Amazingren]

Hey @wyhsirius,
I was training the model on 4gpus, Have you met the following problem:

1. When I directly train start from 0,
   I can use batch_size=32 to train the model without any problem,
   

2. However, when I want to train the model with --resume_ckpt, it shows like below, and I can just use very small batch size to avoid the out of memory problem :
   

I would appreciate it if you can share me some suggestion to solve this problem~

Bests,

[Amazingren]

Hey guys, if you have the same problem. just change

 ckpt = torch.load(resume_ckpt)

to

 ckpt = torch.load(resume_ckpt, map_location='cpu')

in the trainer.py file.