Inaccurate Predictions and Inconsistencies in opal23_headpose_test.py

[jayong-sv]

Hello,
I recently renamed the file "example.tif" to "example_bak.tif" to exclude its annotation file (example.json) and ran the script "opal23_headpose_test.py". However, I encountered significant variations in the predicted values depending on the dataset and rotation mode, resulting in inaccurate predictions.

I observed the following:

• The predicted values varied substantially based on the dataset used.
• Different rotation modes led to inconsistent results.
• The overall accuracy of the predictions was poor.

I am seeking guidance on how to obtain more accurate and consistent predictions from the "opal23_headpose_test.py" script. Any suggestions or insights into improving the prediction accuracy would be greatly appreciated.

• 300wlp_euler

• 300wlp_6d

• panoptic_6d

• panoptic_6d_opal

• panoptic_euler

[alejandro-cobo]

Hello, thank you for your interest.

If I understood correctly, you are trying to call opal23_headpose_test.py without providing the example.json annotations, right?
Since the model needs bounding box annotations to crop the face, I assume you downloaded and configured this face detection repository to get the bounding box annotations. The code defaults to this repo if the JSON file is not found.

As for the wrong predictions you are getting, the models trained on 300W-LP should output the correct predictions. I could not reproduce your results even without the example.json annotations. The models trained on CMU Panoptic, however, won't work on in-the-wild images because the dataset only includes one specific type of background / illumination.

Let me know if this answered your questions.

[jayong-sv]

Thank you for your response!
Yes, I used the SSD detector by default. The 6D predicted values for 300W-LP are accurate, but the 300W-LP Euler predicted values differ from the 300W-LP 6D values. I noted that Panoptic may not perform well on in-the-wild images. Thanks.

[alejandro-cobo]

That's strange, I get the following results using SSD detector and 300wlp_euler model:

And with 300wlp_6d:

Maybe you have an old version of the repo?
Have you tried running git pull on opal23_headpose and images_framework repos?

[jayong-sv]

I set up opal23_headpose and images_framework today, so they should be the latest versions. I really appreciate you taking the time to reproduce the test for me. Given the results, it seems I need to investigate further to determine if there are any issues with my code (I added SSD, images_framework, and updated some paths). Thanks again for your help!

[GKG1312]

Hi @jayong-sv @alejandro-cobo ,
I am trying to replicate the work to get head pose for custom images. I am getting confused at many steps. If any of you can share more details about the implementation it will be very helpful. After setting up this repo with image_framework, while running for custom images, I am getting stucked at image annotation portion, as there is nothing in detection.ssd16_detection.src. Do I need to add the repo for ssd16 there? And suppose I have landmarks from my own detection method, How to use them?

[alejandro-cobo]

For custom data, you need to provide a bounding box of the face in the image, either by using a face detector such as SSD or by providing your own annotations.
For the detector, just run git clone https://github.com/pcr-upm/ssd16_detection.git inside images_framework/detection.
If you have landmark annotations you can use the bounding box of those landmarks in the format (x_min, y_min, x_max, y_max). Check out the test code to see how to add your annotations.

[GKG1312]

@alejandro-cobo
Thanks. I am able to run it now. I was testing it for AFLW2000 but could not figure out where to pass the data or directory. Can you help me with the command?
I am using following command:
opal23_headpose_aflw2000.py --anns-file .\alignment\opal23_headpose\annotations\aflw2000_ann.txt --alignment opal23_headpose --database 300wlp --gpu 0 --rotation-mode euler --target-dist 1.6
and getting following error:
ValueError: Database does not exist

[alejandro-cobo]

For the code to correctly import the modules, you should run the command from the parent directory where the images_framework repo is located:
python images_framework/alignment/opal23_headpose/test/opal23_headpose_aflw2000.py --anns-file images_framework/alignment/opal23_headpose/annotations/aflw2000_ann.txt --database 300wlp --gpu 0 --rotation-mode euler

[GKG1312]

For the code to correctly import the modules, you should run the command from the parent directory where the images_framework repo is located: python images_framework/alignment/opal23_headpose/test/opal23_headpose_aflw2000.py --anns-file images_framework/alignment/opal23_headpose/annotations/aflw2000_ann.txt --database 300wlp --gpu 0 --rotation-mode euler

Still getting same error: ValueError: Database does not exist.
Command running now:
images_framework/alignment/opal23_headpose/test/opal23_headpose_aflw2000.py --anns-file C:\Users\Girish\Desktop\FF-Projects\images_framework\alignment\opal23_headpose\annotations\aflw2000_ann.txt --alignment opal23_headpose --database 300wlp --gpu 0 --rotation-mode euler --target-dist 1.6

In image_framework\alignment\opal23_headpose\data there is a folder for 300wlp but not for the aflw2000.

[alejandro-cobo]

It seems like it is a problem with Windows, as I am able to replicate your error.
The quickest way to solve the error would be to use Linux instead (with WSL, for example).
Let me know if you are restricted to only using a Windows environment, and we can try to make it work on Windows soon.

[GKG1312]

It seems like it is a problem with Windows, as I am able to replicate your error. The quickest way to solve the error would be to use Linux instead (with WSL, for example). Let me know if you are restricted to only using a Windows environment, and we can try to make it work on Windows soon.

Yes, I am currently restricted to windows only. I will see if I can get my hands on a linux system. If you can provide it for windows , it will be of great help.

I am able to run it for custom images, but can not say how good it is performing as I don't have ground truth values for them.

Also, have you considered it to be converted in onnx?

[alejandro-cobo]

The problem should be fixed with commit fef0520.
However, we have found that the annotations file for AFLW2000 is outdated, so the code will still fail until we update it.
You can check if the ValueError: Database does not exist. error no longer appears after updating your local copy of the repo.

As for ONNX, be may convert it in the future if it makes the models more convenient to use.

[GKG1312]

The problem should be fixed with commit fef0520. However, we have found that the annotations file for AFLW2000 is outdated, so the code will still fail until we update it. You can check if the ValueError: Database does not exist. error no longer appears after updating your local copy of the repo.

As for ONNX, be may convert it in the future if it makes the models more convenient to use.

I made the suggested changes, and the error ValueError: Database does not exist. no longer appear. However, it was looking for images in image_framework\alignment\opal23_headpose\annotations\images folder, so I copied all the files (images and .mat files) of AFLW2000 dataset there. Now I am getting IndexError: list index out of range at line 430 in image_framework\src\dataset.py.

Also, can you share the updated annotation file, if you have it?

[alejandro-cobo]

It is fixed now, I can replicate the same results on Windows and Linux.
Let me know if you encounter any other errors after updating to the latest version of the repo.

[GKG1312]

Thanks @alejandro-cobo , Its working fine now.