What do you think about diffgrad?

Question

hadaev8 opened this issue 4 years ago · 3 comments

This is a greate repo, my respect.

Answer 1 · 2020-08-07T03:30:51.000Z

Thanks.

I haven't read the paper on diffgrad. Abstract looks interesting.

Answer 2 · 2020-08-07T12:37:24.000Z

Also, what do you think about weight decay in style of adamw and gradient norming? Will I break anything with gradient norming?

Answer 3 · 2020-08-07T15:05:31.000Z

It should be good.