Somiao Pinyin: Train your own Chinese Input Method with Seq2seq Model

中文Blog

Personalized Chinese Pinyin Input Method with Seq2seq model

Original code in https://github.com/Kyubyong/neural_chinese_transliterator for research purpose.

This repository intends to experiment with different training data and interactive user inputs, and possibly develop towards a real data-personalized and model-localized Pinyin Input product.

Requrements

Python (>=3.5)
TensorFlow (>=r1.2)
xpinyin (for Chinese pinyin annotation)
distance (for calculating the similarity score between two strings)
tqdm

Usage

Training:

STEP 1. Download Leipzig Chinese Corpus

Extract it and copy zho_news_2007-2009_1M-sentences.txt to data/ folder.

Or use your own Chinese Corpus with the same format.
STEP 2. Build a Pinyin-Chinese parallel corpus.

#python3 build_corpus.py

STEP 3. Run prepro.py to make vocabulary and training data.

#python3 prepro.py

STEP 4. Adjust hyperparameters in hyperparams.py if necessary.
STEP 5. Train the model

#python3 train.py

Inference with command line input:

For command line input testing, run:

python3 eval.py

You may change the main function name to use the original testing data evaluation.

Testing with pre-trained models:

Download the pre-trained model from blog, unzip it to generate /log and /data.

Remember to overwrite the pickle files in /data with the pre-trained model data.

Then run for command line input testing:

python3 eval.py

Sample Results

Model is trained from Chinese News in 2007-2009. So many now common Chinese sayings are not learned.

请输入测试拼音：nihao
你好

请输入测试拼音：chenggongle
成功了

请输入测试拼音：wolegequ
我了个曲

请输入测试拼音：taibangla
太棒啦

请输入测试拼音：dacolehuizenmeyang
打破了会怎么样

请输入测试拼音：pujinghehujintaotongdianhua
普京和胡锦涛通电话

请输入测试拼音：xiangbuqilaishinianqianfashengleshenme
想不起来十年前发生了什么

请输入测试拼音：meiguohongzhawomenzainansilafudedashiguan
美国轰炸我们在南斯拉夫的大事馆

请输入测试拼音：liudehuanageshihouhaonianqing
刘德华那个时候好年轻

请输入测试拼音：shishihouxunlianyixiabilibilideyuliaole
是时候训练一下比例比例的预料了

TODOLIST

Pretrained models on different contexts
Model selection for using different models while input different things (chatting? writing scientific papers? etc...)
Function to record LOCALLY what user has input as personalized corpus
User Interface
...

crownpku/Somiao-Pinyin