guillaume-be/SentencePiece-Rust-example

Supporting example for "A Rust SentencePiece implementation"

Implementation of SentencePiece in Rust

This is the supporting code for the article A Rust SentencePiece implementation. Please note that more complete implementation of teh SentencePiece encoding algorithm is available in the rust-tokenizer crate, the code provided in this repository is designed for educational purpose.