N-Grams for novel generation.

This project is a tiny implementation of slp4:N-grams.I wrote some chinese notes for that article:N-Grams模型初探.

About the project

There are 2 files in project: one is for English version of novel generation, the other is for Chinese. I used some simple ways when constructing my N-Grams model, so for further research you may need:

a better trick of smoothing/discount.
use back-off method for high-dimension of N.
validation set, test set. and use perplexity to measure your model.
process unknown words.

English novel generation

python ngram-en.py

example

The pre-words we set:

There was an edge to this

The next 50 words generated from Game of Thrones 01 based on 3-grams is:

There was an edge to this . if i have a name to make the seas rise and saw her first man
was more cunning than she had never known defeat . " i am a sworn sword , and a third of those creatures ,
whatever it takes . i have a name ? oh

Chinese novel generation

pip install jieba
python ngram-cn.py

Generation sample