Training convergence
lvaleriu opened this issue · 8 comments
I'm trying to train the crnn network using latest keras on 2 languages (English and French) on a "youtube spoken" dataset. But it seems the validation accuracy (and not only) blocks at 0.5.
Could you give me some advices about that? I'd like to share in fact some trained models using the latest keras version for the different models you've implemented.
Thanks again!
`
bidirectional_1 (Bidirection (None, 512) 1574912
dense_1 (Dense) (None, 2) 1026
Total params: 8,444,418
Trainable params: 8,439,938
Non-trainable params: 4,480
None
WARNING:tensorflow:Variable *= will be deprecated. Use variable.assign_mul if you want assignment to the variable value or 'x = x * y' if you want a new python Tensor object.
Epoch 1/50
16384/16384 [==============================] - 6176s 377ms/step - loss: 0.1435 - acc: 0.9892 - recall: 0.9999 - precision: 0.5000 - val_loss: 1.6510 - val_acc: 0.6196 - val_recall: 1.0000 - val_precision: 0.5000
Epoch 00001: val_acc improved from -inf to 0.61963, saving model to logs/2018-10-12-02-34-27/weights.01.model
Epoch 2/50
16384/16384 [==============================] - 6156s 376ms/step - loss: 0.0580 - acc: 0.9955 - recall: 1.0000 - precision: 0.5000 - val_loss: 4.5258 - val_acc: 0.5111 - val_recall: 1.0000 - val_precision: 0.5000
Epoch 00002: val_acc did not improve from 0.61963
Epoch 3/50
16384/16384 [==============================] - 6152s 375ms/step - loss: 0.0390 - acc: 0.9964 - recall: 1.0000 - precision: 0.5000 - val_loss: 4.0108 - val_acc: 0.5033 - val_recall: 1.0000 - val_precision: 0.5000
`
hmm, looks like overfitting to the training data... did you try the models on some samples and look what it predicts?
I've tested with some extracted audio waves from youtube videos and it works pretty well for english.
does it work on French at all?
Yes, on a few random audio waves it seems coherent.
I'll try another train soon and get back to you...
It definitely looks like overfitting on your dataset. I don't know why precision always shows 0.5 but if you look at the accuracy, you can see that the train accuracy improves, while the validation accuracy gets worse. Is your dataset large enough? Which model are you using?
I've tried with topcoder_crnn.py, topcoder_small.py, topcoder_deeper.py, cnn.py & crnn.py. I have a dataset with 2 languages (english & french).
I've managed to obtain a 0.94 validation accuracy on this same dataset using https://github.com/stlong0521/language-detector project but I'd prefer using your keras project (and keras 2). I'm pretty sure I'm doing a stupid thing somewhere.
Is there a way we could chat on a slack channel ? (like https://kerasteam.slack.com)
I can give you a private access to a jupyter notebook server if you are interested to see the setup.