pandodao/botastic

Replace tokenizer-go with tiktoken-go

Closed this issue · 0 comments

b1988970-156c-42a5-9285-5335a5be1068

Starting from GPT-3.5, there are some differences between tokenizer-go with tiktoken in encoding result, tiktoken will be more accurate. There is a native Go package implementation: tiktoken-go