qiqihaer/3DSSD-pytorch

RuntimeError: Expected isFloatingType(grads[i].type().scalarType()) to be true, but got false.

Closed this issue · 2 comments

I use pytorch 1.4.0 torchvision 0.5.0 CUDA 10.1 tensorflow 1.15.0
After starting training, I got error below:

2020-10-22 19:07:30,897 INFO **** Saving models to the path log/2020-10-22_19:07:30.897133 ****
2020-10-22 19:07:30,897 INFO **** Saving configure file in log/2020-10-22_19:07:30.897133 ****
2020-10-22 19:07:30,936 INFO **** Dataset length is 3229 ****
epochs: 0%| | 0/120 [00:00<?, ?it/sWarning: No forward pass information available. Enable detect anomaly during forward pass for more information. (print_stack at /tmp/pip-req-build-rz55_vgo/torch/csrc/autograd/python_anomaly_mode.cpp:40)
epochs: 0%| | 0/120 [00:31<?, ?it/s]
Traceback (most recent call last):
File "lib/core/trainer.py", line 190, in
cur_trainer.train() # train !!!!!!!!!!!!!!
File "lib/core/trainer.py", line 170, in train
accumulated_iter=accumulated_iter, tbar=tbar, leave_pbar=(cur_epoch + 1 == self.total_epochs)
File "lib/core/trainer.py", line 131, in train_one_epoch
total_loss.backward()
File "/home/ubuntu/anaconda3/envs/shuyh-3dssd2/lib/python3.7/site-packages/torch/tensor.py", line 195, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/ubuntu/anaconda3/envs/shuyh-3dssd2/lib/python3.7/site-packages/torch/autograd/init.py", line 99, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: Expected isFloatingType(grads[i].type().scalarType()) to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)

Try to use pytorch1.1

Try to use pytorch1.1.0