tamarott/SinGAN

Error Encountered

Closed this issue · 7 comments

Traceback (most recent call last):
File "main_train.py", line 29, in
train(opt, Gs, Zs, reals, NoiseAmp)
File "/gdrive/My Drive/SinGAN/SinGAN/training.py", line 40, in train
z_curr,in_s,G_curr = train_single_scale(D_curr,G_curr,reals,Gs,Zs,in_s,NoiseAmp,opt)
File "/gdrive/My Drive/SinGAN/SinGAN/training.py", line 180, in train_single_scale
errG.backward(retain_graph=True)
File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 198, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/usr/local/lib/python3.6/dist-packages/torch/autograd/init.py", line 100, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3, 32, 3, 3]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Running for mode=random_sample generates the following error

Warning: Error detected in CudnnConvolutionBackward. Traceback of forward call that caused the error:
File "main_train.py", line 29, in
train(opt, Gs, Zs, reals, NoiseAmp)
File "/gdrive/My Drive/SinGAN/SinGAN/training.py", line 41, in train
z_curr,in_s,G_curr = train_single_scale(D_curr,G_curr,reals,Gs,Zs,in_s,NoiseAmp,opt)
File "/gdrive/My Drive/SinGAN/SinGAN/training.py", line 158, in train_single_scale
fake = netG(noise.detach(),prev)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/gdrive/My Drive/SinGAN/SinGAN/models.py", line 60, in forward
x = self.tail(x)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/container.py", line 100, in forward
input = module(input)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 349, in forward
return self._conv_forward(input, self.weight)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 346, in _conv_forward
self.padding, self.dilation, self.groups)
(print_stack at /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:60)
Traceback (most recent call last):
File "main_train.py", line 29, in
train(opt, Gs, Zs, reals, NoiseAmp)
File "/gdrive/My Drive/SinGAN/SinGAN/training.py", line 41, in train
z_curr,in_s,G_curr = train_single_scale(D_curr,G_curr,reals,Gs,Zs,in_s,NoiseAmp,opt)
File "/gdrive/My Drive/SinGAN/SinGAN/training.py", line 182, in train_single_scale
errG.backward(retain_graph=True)
File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 198, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/usr/local/lib/python3.6/dist-packages/torch/autograd/init.py", line 100, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3, 32, 3, 3]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

I also encountered a similar problem.
The source code hasn't changed from before, right?

I've also encountered the first problem,and i can't solve it LOL.

Hi @Yubanwen @taktak1
I got the same error when training a model, it was solved by install torch==1.4.0 and torchvision==0.5.0.

pip install torch==1.4.0 torchvision==0.5.0

Thanks!@chushan89 I solved this problem by changing the torch and torchvison

Thank you so much.