My lr jumped from 0.01 to 0.0498 without any linear signs.

[lxt160980]

Hello! I'm currently using your LinearWarmup and somehow my lr started with 0.1 and then maintained to be 0.0498 until the warmup period was over. I couldn't find out why and here's part of my code.

model = torch.nn.DataParallel(model).cuda()
// args.lr * args.lrf = 0.05
optimizer = torch.optim.SGD(model.parameters(), args.lr * args.lrf, momentum=args.momentum, weight_decay=args.weight_decay * args.wdf)
lr_scheduler =  torch.optim.lr_scheduler.CosineAnnealingLR(optimizer = optimizer,T_max = 23)
warmup_scheduler = warmup.LinearWarmup(optimizer, warmup_period = 5)
# Inside training using epoch, not iteration
 for i, (input, target) in enumerate(train_loader):
        # measure data loading time
        data_time.update(time.time() - end)

        if args.gpu is not None:
            input = input.cuda(args.gpu, non_blocking=True)
        target = target.cuda(args.gpu, non_blocking=True)

        # compute output
        output = model(input)
        loss = criterion(output, target)


        # measure accuracy and record loss
        acc1, acc5 = accuracy(output, target, topk=(1, 5))
        losses.update(loss.item(), input.size(0))
        top1.update(acc1[0], input.size(0))
        top5.update(acc5[0], input.size(0))

        # compute gradient and do SGD step
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # notice: pypi warmup project
        if i < len(train_loader)-1 and warmup_scheduler is not None:
            with warmup_scheduler.dampening():
                pass
# when the epoch ends...
with warmup_scheduler.dampening():
                    lr_scheduler.step()            

Looking forward to your reply!

[Tony-Y]

You should set warmup_period in iteration, not epoch.

[lxt160980]

Omg that's super helpful! Thank u and have a nice day!