/bpe

Byte Pair Encoding Tokenizer

Primary LanguagePythonMIT LicenseMIT

BPE Tokenizer

A fast Byte Pair Encoding Tokenizer written in Python. Inspired by minBPE.

Requirements: