About the design of Cyclical Annealing Schedule

[speedcell4]

Hi~
have you tried any other kind of annealing schedule? e.g. cyclical cosine or cyclical exponential.
I'm just wondering why you chose this linear one. just for simplicity? or this is actually the best?

[speedcell4]

yes, I know your Figure 7, but you did not show the experimental results of them

[ChunyuanLI]

Thanks for your interests, we did not test cosine or exponential schedules in our experiments.

We choose the linear schedule for its simplicity, and we would like to keep the main signal of the paper more focused and consistent; This does not necessarily means there is a clear winner among linear, cosine or exponential schedules.

In practice, we recommend: simply repeating the monotonic schedule in the previous work multiple times to make it cyclical, no matter it was a linear, cosine or exponential schedule.

[speedcell4]

thanks~