microsoft/promptbench

About Semantic Attacks Against Vicuna

tianshuocong opened this issue · 3 comments

tianshuocong commented

Hi!

Thanks for your nice work!

One small question is that when I visited your demo site, and I choose "Vicuna" + "MNLI" + “Semantic” + “zero-shot task”, the site return nothing. Therefore, I am confused and could you please replenish these adversarial prompts? Many thanks!

Best wishes

pb-issue
Immortalise commented

Hi,

Thank you for bringing this error to our attention! We will update the attack results shortly.

tianshuocong commented

Hi!

Thank you very much!

I also noticed that when I selected model as Vicuna, and selected dataset squad_v2，UN Multi, and math, there are also not adversarial prompts.

Immortalise commented

Yes, Vicuna performs much worse on these datasets, we choose not to evaluate Vicuna on these datasets.