/bpe

Primary LanguagePython

Byte Pair Encoding

This is a naive implementation of Byte Pair Encoding (BPE) for tokenization of text.

How to use

Clone the repository and run example.py to train and tokenize a sample text.

git clone https://github.com/shahad-mahmud/bpe.git

python example.py