Best configs / splits

[urialon]

Hi,
Thank you for sharing this great repository and thorough experimental results.

I wish to play with the existing results and possible improvements to the GNN architectures.
Currently, to reproduce results even for a specific architecture and a specific dataset, I need to run (10 folds) * (72 configurations + 3 re-trainings).

Can you somehow share the best-found configurations for each (model) x (task) x (fold)? Or best-hyperparams that were common across all outer-folds, such that my grid search will be smaller?

Thanks!

[diningphil]

Hi,

We are glad you found our work to be valuable.
To reproduce our results, you should necessarily try all configurations as we did for model selection.

The same goes for any new architecture that could improve the state of the art. Restricting a grid search using hyper-parameters found a posteriori on given validation splits will be harmful (biased) to the final estimate.

That is why we are a bit reluctant to publish the best configurations we obtained when training the models. We want to prevent people from looking for patterns in the selected configurations and restrict the model selection on that basis. That would bias the entire process and bring us back to the "wrong way" of doing things, which we criticized in our paper.

With that premise in mind, we would like to make an exception and help you by providing some of the configurations we found in the 10 different model selections. Please email us (federico.errica@phd.unipi.it and marco.podda@di.unipi.it) so that we can discuss your needs and help you with your research :)

[urialon]

Thanks!
That's OK, I will not make exceptions against science :-)

[urialon]

Thanks again for this great repository!