dotnet/TorchSharp

`Module.to(Device)` will cause warnings because non-leaf tensors `.grad` is accessed (Also for optimizers)

yueyinqiu opened this issue · 6 comments

yueyinqiu commented

That's a bit tricky.

We can't just check param.requires_grad because it might be a non-leaf tensor but retains_grad. I'm also not sure whether there are any other situations.

yueyinqiu commented

Yes there are... We can simply do that in PyTorch:

import torch.nn

x = torch.zeros([], requires_grad=False)
x.grad = torch.zeros([]) + 10
print(x.grad)  # tensor(10.)

So we can never know whether their is a grad before we true access it...

Perhaps the only solution is to add a new C++ method for this. Or find a way to temporarily suppress warnings.

NiklasGustafsson commented

Would it be enough to check tensor.is_leaf before accessing grad in the toEpilog() logic?

  • 👀1
NiklasGustafsson commented

I'm confused -- all the instances of .grad in Module are accessing parameters. How can they be non-leaf tensors?

  • ❤️1
yueyinqiu commented

Hmmm... This issue was created due to #1322 . In that issue, this.conv.weight[..] = nn.Parameter(x.view(1, c1, 1, 1)); was used, which made it a non-leaf tensor.

Well I have just reconsidered about this... I suppose it's a misuse of slicing, and a warning is just what we want in this situation.

NiklasGustafsson commented

So, should #1322 also be closed, then?

  • ❤️1