agaldran/balanced_mixup

Reproduction Issue on GI dataset

Closed this issue · 2 comments

hi7049 commented

Dear author,

I tried to use your code to reproduce on GI dataset. Based on your paper, the performance of Bal-Mxp (a=0.3) is 90.39, 64.76 and 64.07. I strictly follow your settings and use 5 fold test and select median results. But I can only get 90.37, 62.15 and 62.23. The MCC is similar, but B-ACC and F1 are lower about 2 points than your results.

Could you give more detailed steps to reproduce your results. Is there any special hyperparameter I should pay attention to?

By the way, could you kindly provide the trained model checkpoints on GI images.

agaldran commented

Hi!

I am sorry but I ran all these experiments in the computer at my previous job, and I never saved the weights. Regarding the results, I am not sure, but after a quick look at the code, I would make two comments:

  • For the endoscopic image classification experiment with mobilenet, which is the one I assume you refer to, what image size are you using? It seems to me that the default is 512x512 (which is the one I used for DR grading), but it could be that I used 512x640 for the endoscopic experiments. If you re-train with that image size, make sure to also use the same size for testing.
  • It could also be just that your system is different than mine, or software versions, or who knows, deep learning is this ugly. If you have a look at table 4 of the appendix (in the arXiv version), you will see that the endoscopic part was quite noisy, with the min/median/max values for five experiments varying wildly.

If you are planning to compare to the results in the paper, and after trying with the modified image size you do not improve results, I would advice you to just say in your manuscript that you re-ran the experiments with our method using the official implementation, and you got those numbers, no problem with that I think.

Good luck!

Adrian

hi7049 commented

Hi Adrian,

Thank you very much for your reply. I got the results by using 512x512 image size to train the endoscopic image with mobilenet. I will follow your advices and try to use 512x640 for training and testing.