How can I connect previous log file into new one in tensorboard
mohanades opened this issue · 1 comments
mohanades commented
Hi,
I trained my model for 50 epochs. Now I want to continue training for other 15 epochs. so I start training with model.load_from_checkpoint("path")
but I don't know how to show continuation of logs with tensorboard
I thought if I write same path of previous file logs, it continue but it didn't show me the previous logs.
I'd be appreciate if you could help me because I need to see the previous training logs along with the new ones.
Part of source code is:
checkpoint_callback=ModelCheckpoint(dirpath="model",filename="newmodel_1",save_last=True,verbose=True,monitor="val_ce_loss",mode="min")
logger=TensorBoardLogger("training-logs",name="test_1")
model1 = model(config)
model_with_previous = model1.load_from_checkpoint(base_path_ckpt)
trainer = pl.Trainer(
max_steps = max_steps,
max_epochs=N_EPOCHS,
gpus=(1 if torch.cuda.is_available() else 0),
logger=logger,
callbacks=[checkpoint_callback],
)
%load_ext tensorboard
%tensorboard --logdir ./lightning_logs
lanpa commented
Hi, you might need to pass the purge_step
parameter.
Here is an low level API example:
https://github.com/lanpa/tensorboardX/blob/master/examples/demo_purge.py