About the usage of validation set.

Hi,

Really nice work! When I run the code, I find that a validation set is used when training the prognnn model. Is the validation set considered when conducting metattack? If we don't use the validation set to pick the best model, whether will the accuracy drop?

By the way, I came into trouble when I try to reproduce the result of vanilla GCN using code in this repo. For example, I can only achieve a ~50% acc with 0.2 metattack on cora dataset. Could you please give me some tips about the hyperparams used to produce the result reported in the paper (~59%).

Thanks a lot!

Hi, thanks for your interest in our work!

(1) We didn't use the validation set when performing metattack. Please see

Pro-GNN/generate_attack.py

Lines 38 to 39 in 84f1757

    
           idx_train, idx_val, idx_test = data.idx_train, data.idx_val, data.idx_test 
        
           idx_unlabeled = np.union1d(idx_val, idx_test)

(2) If we don't use the validation set to pick the best model, the accuracy of the model will normally drop.

(3) I just looked into this issue and found that (a) in the current deeprobust gcn.py, we used both validation loss and accuracy to select the best model:

            if best_loss_val > loss_val:
                best_loss_val = loss_val
                self.output = output
                weights = deepcopy(self.state_dict())

            if acc_val > best_acc_val:
                best_acc_val = acc_val
                self.output = output
                weights = deepcopy(self.state_dict())

(b) the results reported in the paper only used validation loss (following the original GCN implementation)

            if best_loss_val > loss_val:
                best_loss_val = loss_val
                self.output = output
                weights = deepcopy(self.state_dict())

Hi, thank you for your prompt respond!

Considering that when conducting metattack attack, the label info of validation set is not used, should we use it when we train our models against metattack? I feel really confused about the appropriate experiment setting... :)

We followed the setting in the original metattack paper where they did not use validation set when training the attacker but used validation during defense evaluation. I think it is fine as long as the input graph is the same for all the defense methods.

I checked the repo of original metattack. It seems that they do not use validation during defense evaluation. Please check https://github.com/danielzuegner/gnn-meta-attack/blob/master/demo.ipynb for confirmation.

accuracies_clean = []
for _it in tqdm(range(re_trainings)):
    gcn_before_attack.train(split_train, initialize=True, display=False)
    accuracy_clean = (gcn_before_attack.logits.eval(session=gcn_before_attack.session).argmax(1) == _z_obs[split_unlabeled].mean()
    accuracies_clean.append(accuracy_clean)

Ok, I see. In nettack implementation, they used validation to select the model but in metattack it seems they did not use the validation. You may also try to attack the data using validation set and see how the defense performance change. Anyway, I think it should be fine as long as the input graph is the same for all the defense methods.

Thank you! I have tried to modified the deeprobust gcn.py, but I still cannot reproduce the results with vanilla GCN. Could you please provide the hyperparams you use to train the vanilla GCN. Now I can only achieve a ~54% acc with 0.2 metattack on cora dataset, with latest Deeprobust repo and validation based on val_loss. There is still a 5% margin.

Hi, I've looked into this issue and got similar results (~54% acc). Similar issue happens for citeseer with 0.2 metattack (~55.84% acc in the current deeprobust). Other cases seem consistent with the reported results in the paper. I am not sure about what caused it (maybe different versions of packages).

I would suggest you follow the current deeprobust implementation and rerun the experiments for GCN to get these results to make sure fair comparison. One more thing to note here is, while there seems to be little inconsistency in the GCN performance, the performance of Pro-GNN still aligns well with that reported in the paper.

Thank you for pointing this problem out.

Thanks a lot!

	idx_train, idx_val, idx_test = data.idx_train, data.idx_val, data.idx_test
	idx_unlabeled = np.union1d(idx_val, idx_test)