niieani/gpt-tokenizer

Which encoding is this using?

Closed this issue ยท 3 comments

Thanks for your effort with this fork, it's so much faster. It's not clear to me which encoding is being used, is it cl100k_base or r50k_base?

According to this link, gpt-3-encoder is using r50k_base, but I don't see any mention of it in the library.

This is using r50k_base, but I'm working making it customizable in v2.0.

๐ŸŽ‰ This issue has been resolved in version 2.0.0-beta.1 ๐ŸŽ‰

The release is available on:

Your semantic-release bot ๐Ÿ“ฆ๐Ÿš€

๐ŸŽ‰ This issue has been resolved in version 2.0.0 ๐ŸŽ‰

The release is available on:

Your semantic-release bot ๐Ÿ“ฆ๐Ÿš€