a clone of python tiktoken but for PHP! fast BPE tokeniser for use with OpenAI's models.
Primary LanguagePHPMIT LicenseMIT