Wrong initialization for `dataloader_val` in `train.py`

[withchencheng]

This line for initialization of dataloader_val

According to my understanding, the script train.py is for training on Pascal dataset, so we should pass the first argument for initialization of dataloader_val
as args.benchmark and the 4th argument as 0(int zero instead of str zero)?

My configuration for running train.py is:

    dataloader_val = FSSDataset.build_dataloader(args.benchmark, args.bsz, args.nworker, 0, 'val') # shouldn't be fss  ??

Is my configuration correct?

By the way, is it reasonable to set the number of validation images to 346? Is this number too small to validate our model? Can we find other reference materials?

Thanks for your reply.

[slei109]

"dataloader_val" is to evaluate the model during the training process using the validation set. Since we use the whole PASCAL dataset to train the model, it makes sense to use different data as the validation set. You can also set other datasets as needed. But I don't recommend the validation set of PASCAL because the model has seen all the classes.

The trained model will be evaluated on the four different datasets refer to test.py. It will randomly sample different tasks for the evaluation. So it will be possible to have a lot of permutations to generate enough test samples on a very small dataset.

[withchencheng]

Thanks for your prompt reply!

Since we use the whole PASCAL dataset to train the model, it makes sense to use different data as the validation set.
I don't think it makes sense to validate on target domains in the training phase, because:

1. Validation set needs lots of labels,
2. but labels are very precious in few-shot learning.
   Maybe we should use large amount of labels of the target domain only for final testing?

In train.py, do you use validation sets from different target domains when testing the few shot learning?
Or you just use 'fss' as the validation for all target domains in train.py?

[slei109]

Target domain data is not required during the training stage. We used the FSS validation set for all target domains. Thus, there is no loss for the target domain dataset.