Description of Testing on the GAICD dataset.
geraltFromRivia opened this issue · 6 comments
Hello!
Thank you very much for sharing the code and dataset with the community. I have a questions about how would one go about testing previously published methods on the GAICD. I was wondering if you could possibly describe how do you test the VFN / VEN models on the GAICD dataset. Do you feed take the precomputed crops and then feed them each into the VFN / VEN and then rank order the crop scores?
Thank you very much.
Hello!
Yes, we tested previous methods exactly as your description.
Please also note that those crops in the dataset with annotated score "-2" are discarded, cause they have irregular aspect ratios.
Thank you very much for the prompt reply! Is there a chance you'd be able to post a sample evaluation script for VFN or VEN?
I did notice the crops in the dataset with annotated score "-2" are discarded in the dataset definition. I am inquiring about the method of testing, as when using the pre-trained VEN model provided by the authors, the scores computed by my modified testing script computed an SRCC < 0.2, which is much lower as compared to the one reported in the paper.
Thanks!
I uploaded the testing code here:
https://drive.google.com/file/d/1L502GpdTxFW7INQPGD5lR71uRSk6NJBh/view?usp=sharing
The attached results are obtained from VFN. Sorry for not polishing this code.
You may mess up the order of different crops if you only get an SRCC < 0.2.
The result file is formulated as: image ID, crop ID, predicted score.
Thank you very much. I will have a look at the code.