EderSantana/seya

Compile problem with Recursive Container

zwglory opened this issue · 8 comments

Hi,

I was building a model for sequence to sequence with RNN, and I need an attention from a layer to the layer before the output layer. And the Recursive.add_state() is a very good container for this, but I notice that no model class supply for Recursive in Seya/models.py, that means Recursive cannot be compile() and fit(). Can i use Graph's model to compile recursive?

Thanks!

The changes in the Keras API broke that container... if you are using a new version of Keras, it won't work. We started updating the other classes and we will that one soon. But if you want to take at look and help us out with the fix, that would be much appreciated.

The API change I'm talking about is the shape inference one. Are you familiar with that?

Sorry about waiting, and thank you for your invitation, but I am just a newer of Keras, I wish someday I can help.

"The changes in the Keras API broke that container... if you are using a new version of Keras, it won't work. "

The Keras I used is 0.1.3, does the Recursive work in that? If I want to use the Recursive container, how should I do except to build a new API?

if you are using a previous version of Keras, than you should try to install a previous version of Seya as
pip install git+https://github.com/edersantana/seya.git@1d2d58f013f7868735

If you go to the test folder, there as an example on how to use the container: https://github.com/EderSantana/seya/blob/master/tests/test_containers.py#L71-L83

Yes, I have found the example, the container is awesome. Here is my code, but the last two lines(compile() and fit()) didn't work for recursive, and I have not found a model for this, just like class Sequential(Model, containers.Sequential) and class Graph(Model, containers.Graph)
in model.py.

Thank a lot!

RNN_FUN = recurrent.GRU
model = Recursive(return_sequences=True)
model.add_input('input', ndim=2)  
model.add_node(Embedding(vocab_size, EMBED_HIDDEN_SIZE, mask_zero=True),
               name='EL',
               input='input')  
model.add_node(RNN_FUN(EMBED_HIDDEN_SIZE, HIDDEN_SIZE, return_sequences=True),
               name='HL_1',
               input='EL')     
model.add_state('h', dim=HIDDEN_SIZE)   
model.add_node(RNN_FUN(HIDDEN_SIZE, HIDDEN_SIZE, return_sequences=True),
               name='HL_2',
               inputs=['HL_1','h'],
               merge_mode='attention',
               return_state='h')    
model.add_node(TimeDistributedDense(HIDDEN_SIZE, vocab_size, activation='softmax'),
               name='output',
               input='HL_2',
               create_output=True)
model.compile(optimizer='adam', loss='categorical_crossentropy')
model.fit(X, y, batch_size=BATCH_SIZE, nb_epoch=EPOCHS, verbose=1,  validation_split=0.05, show_accuracy=True)

what was the error? merge_mode='attention'? Is this something new in keras?

attention is alignment model devised in NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE by Bahdanau, that I want to implement as a merge_mode in add_node.

The problem is there is no compile and fit in seya/models.py for Recursive,:

model = Recursive(return_sequences=True)
    ...
model.compile(optimizer='adam', loss='categorical_crossentropy')
model.fit(X, y, batch_size=BATCH_SIZE, nb_epoch=EPOCHS, verbose=1,  validation_split=0.05, show_accuracy=True)

I don't know how to train Recursive model and predict with the model without implement the model class for Recursive. And that will be a hard way for me. So I wish there is a solution, thanks.

I think you have to add the container as a node of a Graph.

OK, I will have a try.