Python API: deprecate mode parameter of Tokenizer.tokenize
eiennohito opened this issue · 1 comments
eiennohito commented
The correct way to tokenize with different mode is to create a (new) Tokenizer with that mode
eiennohito commented
Let's keep it around, but somehow try to communicate that the better way tokenize with a single mode is to pass it on the tokenizer creation