karpathy/nn-zero-to-hero

difference on calculating loss for count-matrix

M-Amrollahi opened this issue · 0 comments

When I tried myself to re-implement the code for matrix count, I noticed that there is slight difference between my nll loss and the one in notebook.

Here is what we have in video: -nll = 559891.75

but here what I got: -nll = 559873.5915061831
The difference is about 18.

After reviewing both codes, I found out that the issue is with .item(). It converts the tensor to python number.
In the notebook, this has been calculated as logprob:

log_likelihood += logprob

But I have calculated this:

nll += logprob.item()

Here is example:

print(torch.log(prob).item())
print(torch.log(prob))

>>> -1.4469189643859863
>>> tensor(-1.4469)

In pytorch item, there is nothing mentioned about this.