lucidrains/local-attention

A bug of torch.arange for long sequence with fp16 data type

renll opened this issue 2 years ago · 1 comments

renll commented 2 years ago

pytorch/pytorch#81926

This line should use dtype=torch.long to support long sequence:

local-attention/local_attention/local_attention.py

Line 117 in d30ad4a

ticker = torch.arange(t, device=device, dtype=dtype)[None, :]

lucidrains commented 2 years ago

@renll thanks for identifying this bug!