/CJK-Tokenizer

CJKTokenizer is designed for Chinese, Japanese, and Korean languages. The tokens returned are every two adjacent characters with overlap match.

Primary LanguageJavaScript

Stargazers