abhishekkrthakur/tez

Saving after training an epoch

25icecreamflavors opened this issue · 3 comments

How to save the model after each epoch training? I use fit method for 5 epochs and do not really understand hot to save after each one. not only after the last one.

you can use the EarlyStopping callback and save on each epoch where there is an improvement. here is an example: https://github.com/abhishekkrthakur/tez/blob/main/examples/image/digit_recognizer.py#L175

I'm not sure, will this stop training if there is no improvement for next epoch (I need to continue training anyway)? And do this function saves a model after the whole epoch, not somewhere in the middle right?

You can create a custom callback and use model.save() to save the model at each epoch or if you want to save some specific epochs, that can also be done and that would help in saving model after each epoch, You can use something similar to below code, it maynot perfect but you can create a custom callback similarly

`
class CustomSaver(tez.callbacks.Callback):

    def on_epoch_end(self, epoch, logs={}):

        if epoch <=20:  # or save after some epoch, upto 20

            self.model.save("model_{}.hd5".format(epoch))

`