karpathy/llama2.c

tok512.model adding a token at start.

vish-pr opened this issue · 0 comments

To reproduce:
sp_mode.Encode('R') -> [410, 461]
sp_model.Encode(sp_model.Decode(461)) -> [410, 461]

I am not able to understand this behaviour, not related to bos, eos, unk