Work in progress.
Installation from git requires rustc and cargo to be available and and the wasm32-wasi target to be installed.
Systemcluster/tokenizer
General tokenizer library for the Web and Node. Supports Huggingface and Tiktoken formats
RustBSD-2-Clause