Cannot replicate the result in the multimodal_tutorials
Closed this issue · 3 comments
Hi,
We tried to replicate the results in the multimodal_tutorials, but the spatial clustering result was not as good as the one in the tutorial. Would you provide some suggestions? Please find the model evaluation below. Thanks!
The clustering results from our end,
Result from the tutorial,
Training outputs,
Stopping early: metric has not improved more than 0.0 in the last 8 epochs.
If the early stopping criterion is too strong, please instantiate it with different parameters in the train method.
Model training finished after 96 min 38 sec.
Using best model state, which was in epoch 38.
--- MODEL EVALUATION ---
val AUROC score: 0.9870
val AUPRC score: 0.9787
val best accuracy score: 0.9578
val best F1 score: 0.9589
val target rna MSE score: 6.1840
val source rna MSE score: 1.6078
val target atac MSE score: 0.3834
val source atac MSE score: 0.1222
Hi @dyxmvp,
Did you use the same hyperparameters as in the tutorial? Looking at your evaluation, it looks to me that the model finished quite early (38 epochs) and the MSE scores seem quite high for this dataset. Maybe the model ran into an unlucky optimization.
Yes, I used the notebook provided in the tutorial, and the same hyperparameters as in the tutorial. But I could not get the same results.
This was related to a bug in the random seed configuration. We fixed it now.