/Transformer-ocr

Handwritten text recognition using transformers.

Primary LanguageJupyter NotebookThe UnlicenseUnlicense

Handwritten Text Recognition (HTR) system implemented using Pytorch and trained on the Bentham/IAM/Rimes/Saint Gall/Washington offline HTR datasets. This Neural Network model recognizes the text contained in the images of segmented texts lines.

Data pre-processing is totally based on this awesome repository of handwritten text recognition. Data partitioning (train, validation, test) was performed by following the methodology of each dataset.

Model building is done using the transformer architecture. Recentely facebook research realeased a paper where, they used transformer for object detection. I made few changes to their model so that it could be run on text recognition.

Tutorial (Google Colab/Drive)

A Jupyter Notebook is available for demo, check out the tutorial on Google Colab/Drive.

Datasets supported

a. Bentham

b. IAM

c. Rimes

d. Saint Gall

e. Washington

Requirements

  • Python 3.6
  • OpenCV 4.x
  • editdistance
  • Pytorch 1.5

Command line arguments

  • --source: dataset/model name (bentham, iam, rimes, saintgall, washington)
  • --transform: transform dataset to the HDF5 file
  • --image: prediction on a single image with the source parameter
  • --train: train model using the source argument
  • --test: evaluate and predict model using the source argument
  • --norm_accentuation: discard accentuation marks in the evaluation
  • --norm_punctuation: discard punctuation marks in the evaluation
  • --epochs: number of epochs
  • --batch_size: number of the size of each batch
  • --lr: Learning rate

Notes:

  • Model used is from DETR(facebook research) notebook but in there paper they perfromed few more steps.
  • For improving the results few more things can be done:
    • Using the warmup steps
    • Using sine positional encodings for image vector.
    • Trying more FC layers before output.
    • Trying different parameters of Transformer.
    • Trying different backbone model for getting feature vector of image.
  • Training took ~20 hrs on google colab. where as arthurflor model can be trained in ~8hrs.
  • Word error rate is 15% less when compared to Arthur's model on bentham dataset.
  • Purpose of this project was to showcase the power of Transformer ie: You can use them anywhere.