Running but no checkpoints saved

[celticsha]

Sometimes this runs beautifully on my pc. Other times, it seems to run but doesn't save anything to the checkpoints directory.

[victorchall]

By default the yamls provided will save a checkpoint at the end of every epoch. Are you getting to the end of at least one epoch? Should be when the main steps bar fills...

It may take a while if your repeats and number of training images is large. You can get checkpoints more frequently by reducing repeats in the config YAML. Details are in the readme.

[celticsha]

Thank you for getting back to me. Possibly not getting to the end of the first epoch. I've taken a look at the config file and I could see it's already set to save a file per epoch, but what do I need to tweak if it's not getting that far? I tried increasing steps but ran out of memory.

…

On Sat, 19 Nov 2022, 07:27 Victor Hall, ***@***.***> wrote: By default the yamls provided will save a checkpoint at the end of every epoch. Are you getting to the end of at least one epoch? Should be when the main steps bar fills... It may take a while if your repeats and number of training images is large. You can get checkpoints more frequently by reducing repeats in the config YAML. Details are in the readme. — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A4IJ7PBBJVLJQADIJR5PJ33WJB6ODANCNFSM6AAAAAASEYH6YQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[victorchall]

The new code push in the last 12 hours will now lower system memory use, it should help quite a bit because the CKPT saving takes a lot of system ram, using either RAM or virtual memory/pagefile. Hopefully this will get rid of quite a few of the problems people have on runpods when training thousands of images.

[victorchall]

Open a new issue if you still have issues on the latest code. Full dump of your log will be helpful.