banctilrobitaille/torch-vectorized

RuntimeError: Function 'AcosBackward' returned nan values in its 0th output.

Opened this issue · 1 comments

Getting this error during training. I use a symmetric matrix, overloaded the diagonal with epsilon as well, but still getting this error. Any idea what it might be from?

Hello @metahexane , thanks for reporting this issue. This is probably due to the undefined derivatives of the acos function when the input values are at its domain boundary [-1,1]. I will fix the issue in the following days (I'm pretty busy at the moment). Basically, I will clamp the input values of the acos to be within [-1 + eps, 1 - eps] using the clamp function of Pytorch : https://pytorch.org/docs/stable/generated/torch.clamp.html.