tmbdev/clstm

Implementing clstm's network architecture in Keras

Zerithious opened this issue · 10 comments

Zerithious commented

I wish to implement clstm's network architecture in Keras, to facilitate extension and modifications to the network, and also to add gpu support.

I'm pretty much a newbie to NNs, and I'm trying to wrap my head around the code and the inner workings of clstm. It would greatly help my effort if someone can answer these questions:

  1. What is the exact architecture/topology of the network implemented in clstm? (I mean, what layers, what do they consist of and in which order).

  2. How does it encode the images to be used as input to the network? Is it just the raw pixels of the image with varying width/height?

  3. What other things would I need to know in order to implement this in Keras or a similar framework like tensorflow?

zuphilip commented

@tmbdev Tom, maybe you can share some insights for this questions. Moreover, I think you are already doing something with tensorflow.

amitdo commented

How does it encode the images to be used as input to the network?
Is it just the raw pixels of the image with varying width/height?

Yes. it takes raw pixels as input. The width is always 1 and the height is fixed - by default to 48.

amitdo commented

You might want to look at ocropy's lstm.py code, because I think it will be easier to understand. It was written by the same author who wrote clstm.

mittagessen commented

Keras already implements/utilizes LSTM nets (the network architecture) and Connectionist Temporal Classification (the loss function used for training) from both theano and tensorflow. See keras-team/keras#3436 for more details.

The only thing that's not part of keras is the preprocessing of line images, chiefly dewarping (see lineest.py) whose utility is somewhat questionable as I trained perfectly working models without it.

  • 👍2
tmbdev commented

I already have an LSTM-based OCR for TensorFlow that works like CLSTM, but TensorFlow (and by extension, Keras) support for LSTMs is still less than ideal.

Zerithious commented

@tmbdev Can you link to the repo containing the tensorflow code? Did you mean this?

tmbdev commented

No, that's a multidimensional add-on to TensorFlow. The LSTM code is experimental and in a bunch of iPython notebooks. I'm working on a Torch version right now, which I think will work better.

wanghaisheng commented

@tmbdev any progress ?

rayush7 commented

@tmbdev I am also interested in torch or tensorflow/keras based implementation of clstm. Is there any update from your end?

chinakook commented

A tensorflow implementation of popular library CLSTM and OCROPY line reader
https://github.com/ferjad/tlstm
Does anyone want to modify the repository to the cudnnLSTM or cudnnPersistentRNN implementation, by which the training could speed up about 6x to 10x times ?