WorksApplications/sudachi.rs

Python API: deprecate mode parameter of Tokenizer.tokenize

eiennohito opened this issue · 1 comments

The correct way to tokenize with different mode is to create a (new) Tokenizer with that mode

Let's keep it around, but somehow try to communicate that the better way tokenize with a single mode is to pass it on the tokenizer creation