tiktoken-go

Pinned Repositories

tokenizer
Pure Go implementation of OpenAI's tiktoken tokenizer
Language:Go275 4 317

tiktoken-go's Repositories

tiktoken-go/tokenizer
Pure Go implementation of OpenAI's tiktoken tokenizer
Language:Go275 4 317