Replace tokenizer-go with tiktoken-go
Closed this issue · 0 comments
xwjdsh commented
Starting from GPT-3.5, there are some differences between tokenizer-go with tiktoken in encoding result, tiktoken will be more accurate. There is a native Go package implementation: tiktoken-go