Monitor Localization Accuracy using Intersection Over Union test while training .

[stavBodik]

Hi , I have written code for saving events to TensorBoard file in order to monitor and test localization accuracy while training.

I am using Keras backend, there is option to make custom function and pass it while compiling the model. But the problem is that its hard to debug it because its runs on the GPU .
https://keras.io/metrics/

So Actually I decided not to use this approach and call my own function directly for making averages on each batch end , which predicts and makes IOU test.
Then when epoch ends ,I use on_epoch_end function of my callbacks to write the final average log files.

In order to perform the test I am using y_true array( which is all the prior default boxes then I filter from it only the ground truth positive boxes) and images for the corresponding epoch.

But unfortunately this array is encoded while the assign boxes step and when I am trying to decode it I am getting garbage, for each bounding box the result is an vector which is [ 1, 0, 0, 0 ].

The decode,encode functions I am using are taken from ssd_utils.py , class BBoxUtility .

I attach my test function and the utils code, may some one look at tell how can I fix it please ?

IOU_test.zip

Thanks !

[sudonto]

Hi @stavBodik ,
Thank you for your code. I actually want to visualize my loss function using tensorboard. How did you do this? Can you tell me how to use your code so that I can see my realtime loss function?

Thank you!

[stavBodik]

Hi @sudonto

In order to monitor loss while training you don't need my code, my code is made for special case which is not supported in tensorboard for complex models .

If you want to monitor loss please read here about monitors and summary 👍
https://www.tensorflow.org/versions/r1.1/get_started/monitors
https://stackoverflow.com/questions/34471563/logging-training-and-validation-loss-in-tensorboard?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa

Good Luck !

[sudonto]

Hi @stavBodik ,

Thank you for pointing me to the useful sites. But, I still have difficulties on how to insert logging and monitoring capabilities in this project. This project uses model.fit_generator() syntax to train the network and I dont know how to insert those capabilities into that syntax. I am completely blank at this point. I really need your help.

And second, my training takes about 1 hour using nvidia tesla k40 to train pascal voc 2007 for 1 epoch. Do you think it is normal?

Thank you.

[stavBodik]

@sudonto model.fit_generator() means that you are using Keras framework for tensorboard (which is almost same as .fit of tensorflow).

In keras in order to monitor loss you must use what called callbacks:

https://keras.io/callbacks/

Please pay attention to this callback :

keras.callbacks.ModelCheckpoint(filepath, monitor='val_loss', verbose=0, save_best_only=False, save_weights_only=False, mode='auto', period=1)

I cannot help you much with this because its to much information please google for tutorials and learn:

http://fizzylogic.nl/2017/05/08/monitor-progress-of-your-keras-based-neural-network-using-tensorboard/

https://stackoverflow.com/questions/42112260/how-do-i-use-the-tensorboard-callback-of-keras?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa

http://parneetk.github.io/blog/neural-networks-in-keras/

https://www.pyimagesearch.com/2017/10/30/how-to-multi-gpu-training-with-keras-python-and-deep-learning/

And second, my training takes about 1 hour using nvidia tesla k40 to train pascal voc 2007 for 1 epoch. Do you think it is normal?

Yes.

Unfortunately I cannot help you any more it is to complicated you must read and learn, wish you a good luck !