Why is warmup better than RAdam?

Question

Why is warmup better than RAdam?

Closed this issue 3 years ago · 3 comments

I've argued here LiyuanLucasLiu/RAdam#62 that if warm up and RAdam are equivalent that using RAdam might be simpler - however, I'd be curious about arguments in favour of warm up presented in this repo and related paper.

What are reasons to choose warm up isntead of RAdam?

Answer 1 · 2021-08-04T14:00:31.000Z

It is because the untuned linear warmup works well and is easy to implement.

Answer 2 · 2021-08-05T16:34:57.000Z

It is because the untuned linear warmup works well and is easy to implement.

But RAdam requires no tuning...doesn't that make it better than warm up?

Answer 3 · 2021-08-06T05:21:28.000Z

The untuned warmup dependent on beta2 requires no tuning too.