Description of Testing on the GAICD dataset.

Question

Description of Testing on the GAICD dataset.

geraltFromRivia opened this issue 5 years ago · 6 comments

Hello!

Thank you very much for sharing the code and dataset with the community. I have a questions about how would one go about testing previously published methods on the GAICD. I was wondering if you could possibly describe how do you test the VFN / VEN models on the GAICD dataset. Do you feed take the precomputed crops and then feed them each into the VFN / VEN and then rank order the crop scores?

Thank you very much.

Answer 1 · 2019-10-28T04:23:18.000Z

Hello!

Yes, we tested previous methods exactly as your description.

Answer 2 · 2019-10-28T04:27:10.000Z

Please also note that those crops in the dataset with annotated score "-2" are discarded, cause they have irregular aspect ratios.

Answer 3 · 2019-10-28T04:53:47.000Z

Thank you very much for the prompt reply! Is there a chance you'd be able to post a sample evaluation script for VFN or VEN?

I did notice the crops in the dataset with annotated score "-2" are discarded in the dataset definition. I am inquiring about the method of testing, as when using the pre-trained VEN model provided by the authors, the scores computed by my modified testing script computed an SRCC < 0.2, which is much lower as compared to the one reported in the paper.

Thanks!

Answer 4 · 2019-10-28T05:23:29.000Z

I uploaded the testing code here:
https://drive.google.com/file/d/1L502GpdTxFW7INQPGD5lR71uRSk6NJBh/view?usp=sharing

The attached results are obtained from VFN. Sorry for not polishing this code.

You may mess up the order of different crops if you only get an SRCC < 0.2.

Answer 5 · 2019-10-28T05:25:36.000Z

The result file is formulated as: image ID, crop ID, predicted score.

Answer 6 · 2019-10-28T05:28:25.000Z

Thank you very much. I will have a look at the code.