Questions about the low benign accuracy on CIFAR-10 and GTSRB dataset of Blind Backdoor

Question

Questions about the low benign accuracy on CIFAR-10 and GTSRB dataset of Blind Backdoor

THUYimingLi opened this issue 3 years ago · 8 comments

Hi, Eugene Bagdasaryan,

Congratulations on the acceptance of your paper `Blind Backdoors in Deep Learning Models' and thanks for the sharing of its codes.

However, when we run your code on CIFAR-10 dataset and GTSRB dataset, we get a very low benign accuracy (CIFAR: BA: 18.24, ASR: 98.64; GTSRB: BA: 5.7, ASR: 100) with the default settings in your codes. (PS: we get satisified results on MNIST (BA: 98.86, ASR: 99.99)). We are not for sure where the problems are or whether you used different settings in the experiments of your paper. Can you kindly help us for this problem?

Besides, we also reproduce your codes in our open-sourced toolbox (https://github.com/THUYimingLi/BackdoorBox/blob/main/core/attacks/Blind.py) based on your codes and we meet the same problem. I would be very grateful if you can also help us to check our reproduced codes.

Best Regard,
Yiming Li

Answer 1 · 2022-02-11T13:38:48.000Z

Hey Yiming, let me double check, I tried to simplify the API and might have broken something.

Answer 2 · 2022-02-13T07:51:18.000Z

Hey Yiming, let me double check, I tried to simplify the API and might have broken something.

Thank you for your helps and looking forward to your results :)

Answer 3 · 2022-06-13T21:17:46.000Z

I think my config for cifar confused pre-trained models, but I think simply using a fixed scale and not MGDA will work better.

Answer 4 · 2022-06-14T07:56:09.000Z

I think my config for cifar confused pre-trained models, but I think simply using a fixed scale and not MGDA will work better.

Thanks for your response. I just want to make sure there is no misunderstanding. The model is trained from the scratch instead of fine-tuning on a pre-trained benign model and we adopt the same settings used in your codes. In this case, why we can not reproduce your results?

Answer 5 · 2022-06-17T20:19:31.000Z

Hey, I actually never tested CIFAR as we did Imagenet in the paper so I didn't check it thoroughly, sorry.

I believe it's fixed now. I added a threshold to start poisoning after normal loss reaches 1.0 (we have a fancier idea in the paper on Figure 10 but I think this one works well). I also updated the parameters for the CIFAR run. Can you please let me know if it fixed your issue.

Answer 6 · 2022-10-28T09:05:34.000Z

I think my config for cifar confused pre-trained models, but I think simply using a fixed scale and not MGDA will work better.

hello, I am also try to reproduce your experiment, can you provide your fixed scale that you get?
Thank you!

Answer 7 · 2022-10-28T17:47:07.000Z

I guess you can just set it to 0.9

Answer 8 · 2022-10-29T05:18:21.000Z

I guess you can just set it to 0.9

Thank you for your fast reply!

Here you mean
clean:backdoor = 1 : 0.9 or clean:backdoor = 0.1 : 0.9 ?
Thank you!