niieani/gpt-tokenizer

Is the algorithm of this repo consistent with openai tiktoken?

Closed this issue · 1 comments

Zwe1 commented

Hi, Is the algorithm of this repo consistent with openai tiktoken?
It's important for us to keep consistency with openai

niieani commented

yes, the behavior should be the same, we're using some of the same unit tests.