Training step cell in jupyter lab not showing progress output
artgadliba opened this issue · 1 comments
Hello!
I have encountered an issue, where I cant see any progress output while training process is calculating. It shows only after execution is complete. Im working in Jupyter Lab run locally on Windows Server 2022.
Any ideas how to fix it?
Same issue here @JoePenna !
I'm locally running the Training cell and my Cuda usage is hovering around 65% with 23.6gb of VRAM being dedicated to training. Also, I can see new last.ckpt and global step sample images being created every 500 steps so I know the training is working - although it seems to be running slower than when I ran it on RunPod but that could just be my pc specifications. (20 minutes per 500 steps vs 12 minutes)
@artgadliba are you still having issues?