How long does it take to train the model?
Closed this issue · 4 comments
Hello！Thanks for your good work ! could you please explain what GPU you used to train models, and how long does it take?
I have the same question, seems like dataset is so big that loading it will take a lot of time, so that training process is too slow.
Hello! The model was trained on 8xV100 GPUs for two days.
Loading the dataset is not slow as we do not load the entire dataset at the beginning of training, but chunks of it while simultaneously training the model through the dataloader.
Hello! The model was trained on 8xV100 GPUs for two days.
Loading the dataset is not slow as we do not load the entire dataset at the beginning of training, but chunks of it while simultaneously training the model through the dataloader.
Another question, the dataset seems very big. There are a total of 6 towns, each town collects 100 episodes, and each episode is about 5G, so the entire dataset has 3T space, right?
Something like ~2TB. But it's not necessary to collect so many episodes actually. We did an ablation with 8x fewer data and there was no performance degradation. (so the dataset size can be divided by 8)
Something else for fast experimentation is to set the carla fps to the lowest value possible (10fps). The current fps is 25, so that would save an additional 2.5x in terms of dataset size.
Basically you could get away with collecting roughly ~ 100GB of data :)