namisan/mt-dnn

Maybe it is a bug in alum?

colourful-tree opened this issue · 3 comments

colourful-tree commented

About the Algorithm 1 in paper, https://arxiv.org/pdf/2004.08994.pdf

image

  • I think the Gadv is uesd to update the noise? But, i saw that Gadv don't add any noise in the code below

mt-dnn/alum/adv_masked_lm.py

Line 100 in 99d1b20

adv_direct = self.adv_project(delta_grad, norm_type=self.args.project_norm_type, eps=self.args.noise_gamma)

  • And the variable adv_direct addad to a embed, which doesn't match the paper i think.
    image

mt-dnn/alum/adv_masked_lm.py

Line 101 in 99d1b20

newembed = embed + adv_direct * self.args.adv_step_size

namisan commented

good catch. actually, there is another bug. I haven't have time to fix it yet. will do it after my vacation.

RyanHuangNLP commented

@namisan is there any update about the bug?

namisan commented

I'll update the code in the weekend.