Loss explodes problem
Opened this issue · 2 comments
Hi~, I try to train a GBDTMO model on a large multi-class dataset 'covtype'. The 'covtype' is from libsvm, this dataset contains 7 labels and 580000+ samples.
As the figures shown below, the loss value decreases normally at the very beginning epochs but starts to rise at epoch 21 and finally reaches a very large value at 99 epoch.
This is the final accuracy.
I checked the output model file and found out that some trees have very large output weights. I think perhaps this is the cause of the loss explosion.
I packed the data and the running script together and you can download it through this share link:
https://www.jianguoyun.com/p/DY1tnQ0QruzbCRj89YUE
Could you please check this problem? I think we can discuss together. :D
Loss explosion may happen when we use cross-entropy for training.
Using larger min_samples and reg_l2 is helpful to avoid this.
For example, you can try min_samples=8 and reg_l2=1.0.
A more straightforward solution is to clip the weights of trees. This is used in XGBoost.
But weights clipping is not implemented in this project currently.
Ok, I will try, thanks for your reply