QtacierP/ISECRET

Experiment setting

Nimbus2002 opened this issue · 5 comments

Nimbus2002 commented

Hello:)
hope you are doing well

I have some questions about the experiment settings.

Did you did the cross-validation for the experiment ?
image

for PSNR and SSIM, the value is like this
image
so I assumed you did a cross-validation, but for FIQA and VSD is just a single value like this
image
so I am kind of confused.

Thanks

QtacierP commented

Sorry for the slow reply, I've been on the Chinese New Year holiday and my response may not be as fast. Sorry if this caused any inconvenience. In regards to your question, the +/- symbol represents the standard deviation of the overall test dataset, so the numbers under SSIM and PSNR all represent mean +/- std. As for the VSD and FIQA sections, it's mostly because of formatting and time constraints, and also because they aren't as commonly used as evaluation metrics as the first two, we only put out the mean. Ideally, FIQA and VSD should also be in the format of mean +/- std.
Anyway, I suggest that you can add the std part in the replication or experimental process, it would be more rigorous. Finally, I wish you a successful and happy 2023 in scientific research and life :)

QtacierP commented

Please note that this is not a cross-validation process. I trained a single model and evaluated its performance on the test set. The +/- symbol represents the standard deviation of the test set results. I am sorry for the confusion in the paper :(.

Nimbus2002 commented

Aha! Hope you enjoy your holiday :)
And there is absolutely no need to be sorry, I am very grateful for your kind reply. 🙂

I have one more question, did you use your default settings (epochs: 200, learning rate: 1e-4 ... etc) for you result in the paper?

I am especially curious about the epoch setting and the stopping criteria you used.

Thank you

QtacierP commented

Yes, we closely followed the hyper-parameter settings specified in the paper and utilized a cosine decay schedule for the learning rate, which proved to be crucial for achieving optimal performance. Rather than using stopping criteria, we selected the model that demonstrated the highest PSNR on the validation set.

Nimbus2002 commented

aha I see thanks!