Comparison Accuracy Tensorflow vs Tensorflow Lite
Natriumpikant opened this issue · 6 comments
Hello together,
Since i feel that the performance of my converted tflite models is really low although i have a mAP of 0,96 in the evaluation of the full model.
Is there a way to get the mAP of the TFlite Model also?
Thanks in advance
Hi @Natriumpikant , good question! It's been on my TO DO list to find an easy way to calculate mAP for a TFLite model. This repository on GitHub looks promising:
https://github.com/Cartucho/mAP
Can you look at that repository and see if you can figure out how to use it?
Hey again! Well, I figured out how to calculate mAP of the TFLite model. I wrote a script, calculate_map_cartucho.py, for calculating mAP and uploaded it to the util_scripts folder.
I'll add it to the Colab notebook next. I'm also planning to make a blog post showing how to calculate mAP using the script.
Hey again! Well, I figured out how to calculate mAP of the TFLite model. I wrote a script, calculate_map_cartucho.py, for calculating mAP and uploaded it to the util_scripts folder.
I'll add it to the Colab notebook next. I'm also planning to make a blog post showing how to calculate mAP using the script.
Thank you so far! Adding to your notebook would be very nice - i dont get at the moment how to create the gt-file and output-file to compare within the script.
But take your time implementing it!
I'm on a roll this weekend! I added mAP calculation for TFLite models to the notebook, see Step 7. Please test it out at your convenience and let me know if you're able to get it to work! You'll have to restart the notebook from scratch, unfortunately.
I'm on a roll this weekend! I added mAP calculation for TFLite models to the notebook, see Step 7. Please test it out at your convenience and let me know if you're able to get it to work! You'll have to restart the notebook from scratch, unfortunately.
We are nearly there. I have a last issue because my classes contain many spaces within the classnames. So i get the following error executing "calculate_map_cartucho.py".
Calculating mAP at 0.50 IoU threshold...
Error: File /content/mAP/input/ground-truth/0_0_img0007.txt in the wrong format.
Expected: <class_name> ['difficult']
Received: Kuerbis gross 263 0 429 577
If you have a <class_name> with spaces between words you should remove them
by running the script "remove_space.py" or "rename_class.py" in the "extra/" folder.
Traceback (most recent call last):
File "calculate_map_cartucho.py", line 97, in
with open('output/output.txt', 'r',) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'output/output.txt'
I already tried to get it fixed with following script - https://github.com/Cartucho/mAP/pull/18/commits/892ca5
009e33f30a50e1b7f841b4cd64374b195b
Unfortunately i did not get it running yet.
Got it running.
I'm on a roll this weekend! I added mAP calculation for TFLite models to the notebook, see Step 7. Please test it out at your convenience and let me know if you're able to get it to work! You'll have to restart the notebook from scratch, unfortunately.
After spending multiple hours of work ( please never insert spaces into your classnames ! ) i got it working.
Thanks for your help - think this is a very promising approach for future benchmarking.