This is a java version of Chinese tokenization descried in BERT, including basic tokenization and wordpiece tokenization.
In production, we usually deploy the BERT related model by tensorflow serving for high performance and flexibility. However, our application may not developed by python. Hence, we have to rewrite the tokenization module.
Just run Preprocess.java, you can get result. Now, it support single and pair sentence both.
Moreover, for Chinese natural language processing, we add full turn to half angle and uppercase to lowercase operation.
Please let me know, if you encounter any problems.