Optional: Tokenizer with n-Grams
Closed this issue · 1 comments
XeniaRieger commented
Optional: Tokenizer with n-Grams
XeniaRieger commented
Some functions like detect language and relevance need simple tokens, not n-grams. Maybe two tokenizers?
Closed this issue · 1 comments
Some functions like detect language and relevance need simple tokens, not n-grams. Maybe two tokenizers?