How to monitor model training in python?

[YouthDream0925]

I tried to monitor model training in python.

def model_training():
    try:
        training = replicate.trainings.create(
            version="stability-ai/sdxl:c221b2b8ef527988fb59bf24a8b97c4561f1c671f73bd389f866bfb27c061316",
            input={
                "input_images": input_images,
            },
            destination=destination
        )

        training.reload()
        print(training.status)
        print("\n".join(training.logs.split("\n")[-10:]))

        return True
    except Exception as e:
        print(str(e))
        return False

But I faced an error : 'NoneType' object has no attribute 'split', What's wrong in my code? I followed sample here

How can I monitor model training process in python?

[mattt]

Hi @YouthDream0925. The reason why you're seeing this error is that training.logs isn't populated at the time you've called split on it. In the code you shared, the training is reloaded immediately after creation, so it's unlikely to have started yet and have its logs property populated.

Here's some example code for how you might monitor the progress of a training job (note that trainings take at least several minutes to complete, and as long as several hours, so a polling interval of less than 30 seconds isn't recommended):

while training.status not in ["succeeded", "failed", "canceled"]:
  if training.logs is not None:
    print("\n".join(training.logs.split("\n")[-10:]))
  time.sleep(30)
    training.reload()