About model accuracies
Closed this issue · 3 comments
Hi!
Have you reported thte accuracies of the mbv3 backbone small and large models like with the Resnet backbone somewhere?
Looked @ your Arxiv reports, but could only find the AFLW2000 and BIWI tests on Resnet backbone on the 6D Rotation Representation For Unconstrained Head Pose Estimation paper.
I would be very interested about the inference/accuracy tradeoff & how it competes with the SynergyNet implementation (mbv1/2 backbone). I can also report my results, but I will not be using exactly the same libraries as used for the training and testing, so the accuracies will decrease a little.
I can say that this model behaviour is pretty much what you could wait from mbv3 model; Little bit better accuracy compared to mbv1/-2 backbone nets, and some bit slower.
The MAE is good overall, but need to do new data, align, ... etc. and re-train the pretrain for final model. Due to my company background I can not report the direct results, but it is good model (accuracy vs. inference tradeoff).
If I remember correctly, the MAE for the small mobilenet was about 4.x for ALFW2000. Pretty fair compared to the number of parameters. We have deployed it on one of our robots and the performance for our application is sufficient. The overall problem is the limited rotation range inside the training data (300W-LP). This is a challenge what we are currently tackling, and soon we will open source further work in this regard.
That is actually really good performance! I got around the same result for a controlled test in BIWI, but for the ALFW2000 the result was not as good, but that is very likely because of the wild setting; modules that do not match the training pipeline. Very nice project. Thank you.