ETM training leading to NaN loss
PearlSikka opened this issue · 3 comments
- OCTIS version: 1.10.4
- Python version: 3.7.13
- Operating System: Windows
Description
I'm running topic model for tweets using ETM model. While training, it led to NaN loss in the first epoch and hence, the training doesn't go further epochs. The ETM model is being trained with default parameters.
model = ETM(num_topics=10) #command run
output = model.train_model(dataset)
Output:
Epoch: 1 .. batch: 20/25 .. LR: 0.005 .. KL_theta: nan .. Rec_loss: nan .. NELBO: nan
Hello,
this is an issue related to the original implementation of ETM. We took the model and integrated into OCTIS. Looking at a related issue in the original repo (adjidieng/ETM#3), it seems that lowering the learning rate could help. The other two parameters (bow_norm and activation_function) are okay by default.
Otherwise you can try using a different model, e.g. CTM seems to work well on short texts as tweets.
Let me know if it helps,
Silvia
Thank you Silvia for your quick response. I tried training ETM with lower learning rate as well but it still shows NaN loss. Maybe I can leverage CTM model. Thanks again!
Okay, then I'll close the issue. Feel free to re-open it or open a new issue if you have other questions.