/sub-word-Tokenizer-test

sub-word Tokenizer test

Primary LanguagePythonApache License 2.0Apache-2.0

sub-word-Tokenizer-test

sub-word Tokenizer test

A little test to know if tokenizers prefer sub-word tokenization over complete word tokens.