how to decide on the num_batches. i was trying to do training for 3 classes with num_batches=100 and batch_size = 16. but the code is not running, it is skipping every if loop in voxnet_train.py. it stops after some time. it is not throwing any error though
ajaysg-zz opened this issue · 2 comments
num_batches = 100 is probably too little batches.
I've set the number of batches to a very large integer (i.e. num_batches = 2147483647)
This means that the training will run almost forever and I will manually have to stop it via the terminal when the accuracy is high enough.
if batch_index and batch_index % 512 == 0:
the training code checks the loss and decreases the learning rate a bit every 512'th batch after the 0th batch.
if batch_index and batch_index % 2048 == 0:
the training code checks the accuracies saves a checkpoint every 2048 batches after the 0'th batch.
You don't want to be evaluating the accuracies and saving too often, because these will slow down the training.
@Vectorized thank u for the help