Training Time
Opened this issue · 1 comments
Hello,
thank you for your great work.
How long did it take you to fully train this model on the ycb dataset?
How long does it roughly take you to train for one epoch?
On my setup GTX1080ti + SSD + Intel Xeon it takes for 1000 samples roughly 1 Minute. Given 100.000 training samples * 5 (to get all object in each frame) / 1000 * 60s = 8.3h for one epoch.
Best regards, Jonas
Hi, so in my case with the dataset on HDD, it's about 3h per epoch (also 1080ti but with i9). I think SSD would be faster. There is a way to increase training speed. Basically, you can pre-process the synthetic data of the YCB dataset by adding the background coco image and save them to a new folder. During training, you can directly load the processed data instead of processing them each time during training.