DreamtaleCore/PnP-GA

How to eliminate the influence of different head poses

Closed this issue · 5 comments

davidyinnan commented

dear sir, I want to know how to eliminate the influence of different head poses. In eyediap, ETH-XGaze and gaze360, there are varieties of head poses. It is mentioned in the paper that "we eliminated the influence of different head poses through rotating the virtual camera and wrapping the images". Does that mean the rotation of head pose of all the three axes are eliminated or just one axe (like MPIIGaze or ETH-XGaze processed)?. Could you give me some example codes or explaination? Thanks very much!

MickeyLLG commented
davidyinnan commented

Yes, we follow the processing method in MPII and ETH-XGaze, which mainly eliminates the rotation round z axis. I would like to provide the souce code on our lab's official website, have fun with it. ^_^ https://phi-ai.buaa.edu.cn/Gazehub/rectification/
…
------------------ 原始邮件 ------------------ 发件人: "shuang @.>; 发送时间: 2022年12月8日(星期四) 晚上10:56 收件人: @.>; 抄送: @.>; 主题: [DreamtaleCore/PnP-GA] How to eliminate the influence of different head poses (Issue #6) dear sir, I want to know how to eliminate the influence of different head poses. In eyediap, ETH-XGaze and gaze360, there are varieties of head poses. It is mentioned in the paper that "we eliminated the influence of different head poses through rotating the virtual camera and wrapping the images". Does that mean the rotation of head pose of all the three axes are eliminated or just one axe (like MPIIGaze or ETH-XGaze processed)?. Could you give me some example codes or explaination? Thanks very much! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.>

Thanks for your early reply! I have tried the code on the website. It works! And I have one more question that it seems there are no camera parameters provided in gaze360. How could I process gaze360? Could I just use the method directly on http://phi-ai.buaa.edu.cn/Gazehub/3D-dataset/#gaze360 or there is another way to process?

davidyinnan commented

关于ETH数据集的使用，在你们iccv2021年的文章中谈到利用MPIIGaze【37】来进行数据处理，"However, the change of head poses will significantly affect the appearance of face image. Thus, we eliminated the influence of different head poses through rotating the virtual camera and wrapping the images."，并额外消除了headpose的影响，我看了你们给我的预处理的源代码(https://phi-ai.buaa.edu.cn/Gazehub/rectification/)，和【37】的原理一样，仅消除了headpose一个轴的影响。ETH数据集有18个机位，除了机位00和机位01是大致正对人脸（基本消除了headpose的影响），其他机位都有明显的head pose，用你们提供的预处理代码并不能像iccv2021文章中说的消除。我想请问一下，你们是否在训练时18个机位的数据都用了（因为同一帧的注视点都是相同的），还是仅用了机位00或机位01的数据呢？否则，你们提供的预处理代码好像不能按照文中说的消除headpose的影响，我十分困惑，请您指点。

MickeyLLG commented

good question^_^
1.首先，训练时ETH18个机位的数据均使用了。
2.“head pose”这一概念并不仅仅是头部旋转，还包含了平移向量(tvec)，尽管【37】的预处理可以认为仅消除了绕z轴的滚转角，但不可忽视该预处理同时消除了平移向量的影响，把所有面部图像均转换为相同的tvec。
3.原文“eliminated the influence of different head poses”这一说法诚如您所说，不够严谨，因为仅消除了滚转角。但也如2所说，其同时消除了滚转角+平移向量的影响。
4.在视线估计任务中，我认为“同时消除了滚转角+平移向量的影响”已经足以称为“消除了headpose的影响”。尽管另外两个轴的旋转角会影响图像的外观，但这是与gaze强相关的信息，且是CNN应该学到的信息。我认为消除了滚转角+平移向量的影响之后，输入图像在gaze任务上的语义已经被处理到统一了，对CNN来说是相同格式的输入。

davidyinnan commented

good question^_^ 1.首先，训练时ETH18个机位的数据均使用了。 2.“head pose”这一概念并不仅仅是头部旋转，还包含了平移向量(tvec)，尽管【37】的预处理可以认为仅消除了绕z轴的滚转角，但不可忽视该预处理同时消除了平移向量的影响，把所有面部图像均转换为相同的tvec。 3.原文“eliminated the influence of different head poses”这一说法诚如您所说，不够严谨，因为仅消除了滚转角。但也如2所说，其同时消除了滚转角+平移向量的影响。 4.在视线估计任务中，我认为“同时消除了滚转角+平移向量的影响”已经足以称为“消除了headpose的影响”。尽管另外两个轴的旋转角会影响图像的外观，但这是与gaze强相关的信息，且是CNN应该学到的信息。我认为消除了滚转角+平移向量的影响之后，输入图像在gaze任务上的语义已经被处理到统一了，对CNN来说是相同格式的输入。

十分感谢您的回复，我现在弄清楚了！