grad is None when upgrading mmdet to a higher version (2.16.0)
Closed this issue · 3 comments
Hey, dragon. Thank you for your work!
I am trying to make ISAL compatible to a higher version of mmdet (e.g., 2.16.0). While other methods work fine, ISAL throws an error in src/test_calc_s_test.py, and the error log is
[ ] 0/1784, elapsed: 0s, ETA:Traceback (most recent call last):
File "src/test_calc_s_test.py", line 146, in <module>
Traceback (most recent call last):
File "src/test_calc_s_test.py", line 146, in <module>
main()
File "src/test_calc_s_test.py", line 131, in main
loss_val_mul_hessian_reverse = multi_gpu_s_test(val_sum_grad, model, params, train_data_loader, img_num=args.img_num)
File "/home/username/Detection/isal/src/utils.py", line 146, in multi_gpu_s_test
main()
File "src/test_calc_s_test.py", line 131, in main
hv = hessian_vec_prod(all_loss, params, h_estimate)
File "/home/username/Detection/isal/src/utils.py", line 203, in hessian_vec_prod
return_grads = [ele.detach() for ele in return_grads]
File "/home/username/Detection/isal/src/utils.py", line 203, in <listcomp>
loss_val_mul_hessian_reverse = multi_gpu_s_test(val_sum_grad, model, params, train_data_loader, img_num=args.img_num)
File "/home/username/Detection/isal/src/utils.py", line 146, in multi_gpu_s_test
return_grads = [ele.detach() for ele in return_grads]
AttributeError: 'NoneType' object has no attribute 'detach'
Traceback (most recent call last):
File "src/test_calc_s_test.py", line 146, in <module>
hv = hessian_vec_prod(all_loss, params, h_estimate)
File "/home/username/Detection/isal/src/utils.py", line 203, in hessian_vec_prod
return_grads = [ele.detach() for ele in return_grads]
File "/home/username/Detection/isal/src/utils.py", line 203, in <listcomp>
return_grads = [ele.detach() for ele in return_grads]
main()
File "src/test_calc_s_test.py", line 131, in main
AttributeError: 'NoneType' object has no attribute 'detach'
loss_val_mul_hessian_reverse = multi_gpu_s_test(val_sum_grad, model, params, train_data_loader, img_num=args.img_num)
File "/home/username/Detection/isal/src/utils.py", line 146, in multi_gpu_s_test
hv = hessian_vec_prod(all_loss, params, h_estimate)
File "/home/username/Detection/isal/src/utils.py", line 203, in hessian_vec_prod
return_grads = [ele.detach() for ele in return_grads]
File "/home/username/Detection/isal/src/utils.py", line 203, in <listcomp>
return_grads = [ele.detach() for ele in return_grads]
AttributeError: 'NoneType' object has no attribute 'detach'
Traceback (most recent call last):
File "src/test_calc_s_test.py", line 146, in <module>
main()
File "src/test_calc_s_test.py", line 131, in main
loss_val_mul_hessian_reverse = multi_gpu_s_test(val_sum_grad, model, params, train_data_loader, img_num=args.img_num)
File "/home/username/Detection/isal/src/utils.py", line 146, in multi_gpu_s_test
hv = hessian_vec_prod(all_loss, params, h_estimate)
File "/home/username/Detection/isal/src/utils.py", line 203, in hessian_vec_prod
return_grads = [ele.detach() for ele in return_grads]
File "/home/username/Detection/isal/src/utils.py", line 203, in <listcomp>
return_grads = [ele.detach() for ele in return_grads]
AttributeError: 'NoneType' object has no attribute 'detach'
It seems that the 2nd derivative of conv_cls is None. Could you help with it?
Thanks in advance!
Hi! Please check the second backprop is successful or not
in file "/home/username/Detection/isal/src/utils.py", line 201, 'return_grads = grad(per_param_grad, params, create_graph=True, allow_unused=True)'.
I will close this issue, if you have further questions, please do not hesitate to reopen it!
@dragonlzm
Thank you and sorry for my late reply lol. I am still stuck here.
During debugging, I found that the first_grads returns all-zeros, which could lead to the None results in the 2nd backprop return_grads.
What could cause this error?