- This repository is only for study, please do not use for commercial.
Theano implementation of LSTM and CTC
- Support GPU accelaration. (Only when the model is complicated enough the GPU acceleration can be seen)
- Support LogCTC, which prevents from overflow issue
- Support batch training, which means that different width of images can be packed into a single mini-batch
- Support prediction alone script, useful to test and release.
- Currently not support CNN feature extraction, as there are plenty framework to do so. Raw pixel features in gray scale are feed directly into LSTM and CTC layers.
Data format
- x is a list of images. The size of each image should have identical height, and the width varies.
- x_mask is a matrix to indicate the valid region of each image. As in a mini-batch, images with different width are packed together, we need extra infomation to store the real width of each image. For example,
[[1, 1, 1, 0, 0], [1, 1, 1, 1, 1]]
represents 2 images, the width of first image is 3 and the second one is 5 - y is a matrix, each row is the corresponding label (sequence of chars) of each image.
- y_clip is a vector, similar as x_mask, however in more compact format. Each element represents the real length of corresponding label.
Installation
Ubuntu:
sudo apt-get update
sudo apt-get install -y libmagickwand-dev python-opencv
pip install -r requirements.txt
Mac OS X:
brew tap homebrew/science
brew install opencv
pip install -r requirements.txt
Add package path to your environment, for example
export PYTHONPATH=/path/to/cnn-lstm-ctc/:$PYTHONPATH
OpenCV3 is needed
Get Data
- Pull all codes and data, including english_sentence.tar.gz (~21MB) in dataset folder
- Extract the files and run get_data.sh
cd /path/to/cnn-lstm-ctc/
cd dataset
tar -zxvf english_sentence.tar.gz
cd ..
sh get_data.sh
Train
cd /path/to/cnn-lstm-ctc/
sh train.sh
A typical validation accuracy curve within 200 epochs, lr=0.01, gamma=0.1, multistep_epoch=[100, 150, 175], cost 28.255 hours. (In this repository only 0.01% data are released due to privacy)
The best result occurs in epoch 188 with 67.152%
Predict
- After training converged and finished, pick the favorite model in snapshot folder, say, 188.pkl.
- Open predict.sh and modify
model_path=snapshot/188.pkl
cd /path/to/cnn-lstm-ctc/
vim predict.sh
sh predict.sh
Others
- The first time to compile the train and val function in train/train.py takes long, maybe 10-15 mins
- For simple english word recognition task, with whole dataset, 92.8% accuracy can be obtained after 200 epoch.