BPE tokenization implemented in Golang 💙
Primary LanguageGoApache License 2.0Apache-2.0
Please refer to tokenize and detokenize APIs for up to date Cohere tokenizers.