IBM/cdfsl-benchmark

Question regrads to enviroment

Closed this issue · 6 comments

zilunzhang commented

Do we have to use PyTorch and python version as below for the competition?

Python 3.5.5

Pytorch 0.4.1

h5py 2.9.0

Thanks

yunhuiguo commented

yes, if you have any problems please let me know.

zilunzhang commented

yes, if you have any problems please let me know.

That would be all for the dependency setting! Thanks. However, I also have some other questions for the competition requirements...

  1. What about the version of cuda and cudnn?
  2. Is there a restriction for gpu amount or type (Telsa/Titan/Geforce etc.) that we are allowed to use?
  3. From another opened issue, it seems that it's okay if we tune the hyper-parameter like batch size/epoch size/embedding size, which will lead to different performance for sure. So, based on my understanding, we can do what we want to the codebase as long as the skeleton of the codebase is kept? (Or we could use another codebase if we make sure (1) training set and test set, (2) way number, (3) shot number, (4) query number, as well as (5) 600 randomly sampled few-shot episodes and 95% CI for accuracy in the evaluation stage follow the requirement?)

Thanks

yunhuiguo commented

Hi @zilunzhang,

  1. For cuda and cudnn, I think there are multiple compatible versions. You can use cuda8.0 or higher versions as long as there are no errors.

  2. It would be better to use the provided codebase to ensure the setting are the same. However, if you want to use another codebase, you need to make sure the settings you pointed out are the same.

zilunzhang commented

Hi @zilunzhang,

  1. For cuda and cudnn, I think there are multiple compatible versions. You can use cuda8.0 or higher versions as long as there are no errors.
  2. It would be better to use the provided codebase to ensure the setting are the same. However, if you want to use another codebase, you need to make sure the settings you pointed out are the same.

Hi @yunhuiguo,

Thanks for the answer!

Is there any other setting that needs to be consistent for the competition besides the 5 settings mentioned above?

Also, just to make sure, the backbone has to be ResNet10 for images with shape (3, 224, 224) for any other codebase?

yunhuiguo commented

Hi @zilunzhang,

  1. Besides the mentioned 5 settings, the data being used should stay the same, i.e., no additional data sources are allowed. Basically, you should follow the standard way for evaluating few-shot learning algorithms.
  2. And you are free to change the backbone and the image resolution.
zilunzhang commented

Hi @zilunzhang,

  1. Besides the mentioned 5 settings, the data being used should stay the same, i.e., no additional data sources are allowed. Basically, you should follow the standard way for evaluating few-shot learning algorithms.
  2. And you are free to change the backbone and the image resolution.

Hi @yunhuiguo,

That's a very clear answer, thank you!