/bpe-example

Supporting code for post on Rust implementation of Byte pair Encoding

Primary LanguageRustApache License 2.0Apache-2.0

Byte pair Encoding implementation examples

This is the supporting code for the article Byte pair Encoding and Data Structures. Please note that more complete implementation of the BPE encoding algorithm is available in the rust-tokenizer crate, the code provided in this repository is designed for educational purpose.