Reproducing the results in the Paper
thegialeo opened this issue · 8 comments
Dear Spyros Gidaris, Praveer Singh and Nikos Komodakis,
i have read your paper "Unsupervised Representation Learning by Predicting Image Rotations" and was impressed by your work and the astonishing results receive by pretraining a "RotNet" on the rotation task and later train classifiers on top of the feature maps.
I have downloaded your code from GitHub and tried to reproduce the values in Table 1 for a RotNet with 4 conv. blocks. However, running "run_cifar10_based_unsupervised_experiments.sh" and altering line 33 and for 'conv1' also line 31 in the config file "CIFAR10_MultLayerClassifier_on_RotNet_NIN4blocks_Conv2_feats.py", i obtain slightly lower values than in the paper especially for the fourth block:
Rotation Task: 93,65 (Running your Code) / ---
ConvBlock1: 84,65 (Running your Code) / 85,07 (Paper)
ConvBlock2: 88,89 (Running your Code) / 89,06 (Paper)
ConvBlock3: 85,88 (Running your Code) / 86,21 (Paper)
ConvBlock4: 54,04 (Running your Code) / 61,73 (Paper)
Are there further things I need to consider before running the code to achieve the results in the paper? I have used a GeForce GPX 1070 to run the experiment.
Excuse me, I am a beginner, I want to ask you a question: the author omitted the detail code of function train_step, evaluation_step, without this part of the code, is not training? Did you complete it? Looking forward to your reply. Thank you @Xenovortex
It is a year ago, so I don't remember in detail. But to my knowledge, those functions are implemented and it should work out of the box. Maybe I didn't exactly understand your question/problem.
Also I did my own implementation of the FeatureRotNet (based on the knowledge provided in the paper), if you need some reference you can have a look at it, but i am pretty sure their original code should work as well.
Sorry, I didn't look at the code carefully before. I have studied it carefully these two days and found that the code can work successfully.Thanks for your reply. @Xenovortex
no problem
Hi, @Xenovortex .
Have you reproduced supervised NIN? I have reproduced supervised NIN as the paper setting, which is 88.23 in my experimental results, but the result of the paper report is 92.80.
Thank you.
Yes, I did reproduced the supervised NIN. With my code (https://github.com/Xenovortex/Implementation-FeatureLearningRotNet), I achieved slightly lower accuracy than in the paper but pretty close:
Total Accuracy: 91.39 %
Accuracy by classes:
plane: 91.60%
car: 95.90%
bird: 87.00%
cat: 83.20%
deer: 92.10%
dog: 85.50%
frog: 94.80%
horse: 93.90%
ship: 94.70%
truck: 95.20%
However, running the code of the authors, I got 92.81%.
Yes, I did reproduced the supervised NIN. With my code (https://github.com/Xenovortex/Implementation-FeatureLearningRotNet), I achieved slightly lower accuracy than in the paper but pretty close:
Total Accuracy: 91.39 %
Accuracy by classes:
plane: 91.60%
car: 95.90%
bird: 87.00%
cat: 83.20%
deer: 92.10%
dog: 85.50%
frog: 94.80%
horse: 93.90%
ship: 94.70%
truck: 95.20%However, running the code of the authors, I got 92.81%.
Thank you for your reply.
The num_stages is 3 in the author code of supervised NIN (that is nine convolutional layers), but num_stages is 4 in the unsupervised NIN architecture (that is twelve convolutional layers), so I am a bit confused. Such unsupervised results and supervised results may be a bit unfair.
I mean, if you think it is an unfair comparison. Then, there is really nothing stopping you from changing num_stages to 3 or 4 respectively and run the authors code to see if it makes a difference.