flutter_tiktoken is a flutter offline package for a fast BPE tokeniser for OpenAI models.
Primary LanguageDartMIT LicenseMIT