Byte2Byte Tokenizer

[loretoparisi]

Implement a "token-free" or tokenization free encoder to work at Unicode/UTF-8 character-level.

Examples

• CANINE, Unicode code points HF, arxiv
• ByT5, UTF-8, HF, arxiv