preddy5/Im2Vec

Default config (emoji.yaml) fails to converge

Closed this issue · 7 comments

Hey, thanks so much for the code release. Really excited about the potential of this project.

I think the default config might have some incorrect hyper parameters? After 300-400 epochs I'm still not seeing the model able to reconstruct the emoji dataset.

Any guidance on what parameters to use would be appreciated. I tried what was suggested in #5 but did not see an improvement.

recons_VectorVAEnLayers_0383

Hey @evancasey
This tends to happen with high learning rates, could you try decreasing the learning rate https://github.com/preddy5/Im2Vec/blob/master/configs/emoji.yaml#L23 to 0.0001 and see if the changes anything.

I'm starting to see better results after reducing LR to 0.0001, however it takes much longer to train than your logged results. Would you suggest tuning LR some more or are there any other hyperparameters that come to mind?

recons_VectorVAEnLayers_0547
After 500 epochs, LR=0.0001

After 2000 epochs it still does not converge

Hey @evancasey I think you might have to tune the LR a bit, could you try 0.0002 maybe, also increasing https://github.com/preddy5/Im2Vec/blob/master/configs/emoji.yaml#L8 the radius helps with the initial part of the optimization maybe try 20. Hopefully, this helps with the optimization.

Same here. Tried everything. Almost 3000 epochs and it still does not converge.
There are 3 logical possibilities:
1 - The code in this repository is not the one used in the paper (i.e. notebooks?)
2 - There is a secret way to tune those parameters with a separate tool/script
3 - The results in the paper are made up
Excluding the third, both the 1st and the 2nd explanations are easy to solve.
You just need to merge/add the source code of the notebook or of the tool with this repo.
Are you still under deadlines? I hope you will find a moment to fix this. This is an important work.
Thanks. 👍

Hey @Emasoft
There are many reasons why the network might not have converged when you train it. 
The repo includes an example training run please use that for debugging your training. For each new training run, the code also creates a copy of the training code in the log directory, so the example training run also contains a backup of the code used for that specific training and trained weights, you can use them for debugging as well. 

Hope that helps.