How to use a customize tokenizer?

Question

How to use a customize tokenizer?

perkfly opened this issue 3 years ago · 1 comments

For east Asian languages such as Chinese, tokenizer can be very important. I notice that tantivy has some third-party tokenizers, how do I use it in lnx?

Answer 1 · 2022-06-01T10:03:03.000Z

Currently, there's very minimal support for CJK language tokenizers within lnx, currently, we only really support Latin languages currently .(although future versions will have this)