陀螺仪稳像代码理解以及相关问题请教
Closed this issue · 15 comments
您好，
首先，感谢您开源的陀螺仪稳像与卷帘快门效应纠正的实现代码，学习以后受益匪浅~
其次，拜读完您的代码实现后，我在下面稍微整理了一下，不知道理解的对不对，如有误还望指出，谢谢~
-
在代码gyro_readings_to_H.py中，对相邻两帧图像对应帧内十个片段之间的所有陀螺仪角速度数据进行欧拉积分，并对计算得到的若干旋转矩阵进行累乘后，得到这两个相邻帧之间的旋转矩阵（共十个），再通过Karpenko论文公式（8）计算出对应单应矩阵，用于卷帘快门效应剔除。
-
在代码gyro_aglinment.py中，根据卷帘快门相机逐行曝光原理，利用第一个步骤中获取到的所有帧内patch块的单应矩阵对前一帧图像进行透视变换，并将变换后图像中patch块的像素值保存到输出图像对应patch块中，依次遍历完所有patch，就完成了卷帘快门畸变效应的剔除。计算单应变换前后相邻帧之间的PSNR值，用于评价两帧的相似程度，PSNR值越大越相似，也说明矫正效果越好。
上述两个实现步骤完成的效果，应该主要是前后相邻帧间的卷帘快门效应剔除吧？防抖中的稳像平滑步骤是否还未提及？
我也看了一下Karpenko的matlab代码（仅仅是看他的代码，未成功跑出效果，看的也有一点一知半解。。。），貌似是对原始陀螺仪角速度施加了一个高斯滤波器，然后分别利用滤波前后的角速度值进行中值积分，累加旋转角度后得到IMU的朝向，再通过线性插值计算出视频帧的旋转朝向。之后在meshwarp.m中进行稳像处理以及帧内卷帘效应畸变矫正。
此外，还想请教您几个问题：
-
在Karpenko的论文中提到，其手机端的稳像系统将相机运动仅建模为旋转运动，而不考虑平移，那在车载稳像这种比较快速的骑行场景中，这个假设是否还成立？或者相机的平移运动是否需要考虑进来？
-
刘老师的meshflow提供了头文件以及cpp实现，若想要运行起来，您是否方便开源一下meshflow用于稳像的demo示例，谢谢~
-
若想要对采集到的视频和IMU数据进行实时稳像处理，平滑算法目前我只知道可以通过卡尔曼滤波器进行相机运动轨迹的实时平滑，其它的不是很了解，后续您是否会开源这方面的实时防抖路径平滑优化代码以供学习，谢谢您~
我接触防抖这块的内容不是很久，基础相对比较薄弱，也是很想花时间学习一下，上面的描述可能也有点冗长，如有打扰，深表歉意，再次感谢您（@lhaippp）以及刘老师优秀的学术工作以及开源贡献~
Hi Clark,
Thx for your attention for our work.
Firstly, your summary seems to be correct in my mind. And for the optimization of stabilization path, we consider to discuss the open-source plan in the near future.
I think the solution of Karpenko et al. seems to be similar to https://github.com/gyroflow/gyroflow, while our methods is slightly different from theirs, cos we leverage the path optimization algorithmn from Bundled Camera Path.
For the rest questions:
- For videos captured during driving, the assumption is ok and we do not need the translational info. We have verified it.
- The demo for meshflow, it might also need permission, I will verify it.
- For the path smoothing algorithm, I recommend you to follow Bundled Camera Path
Lastly, I recommend a repo to you: https://github.com/yaochih/awesome-video-stabilization
Hi,
Thanks for your quick reply and suggestions~ Looking forward to your future open-source work.
Last weekend, I have reviewed the paper and matlab code for Bundled Camera Path. The Liu's code does not include the implementation of path optimization algorithm, but is only used for motion model estimation. Another repo of SuTanTank provides a third party implementation of Liu's paper. The stabilized video doesn't look very obvious.
As I know, the solution of gyroflow you mentioned is based on a post-processing algorithm, which includes forward and reverse pass using quaternion slerp smoothing. So I think it utilizes all the information about camera orientation trajectory (past and future). If I want to re-implement it for online smoothing, do you know how to design it. Give me some ideas, pls.
Thx again.
Hi,
-
''The Liu's code does not include the implementation of path optimization algorithm, but is only used for motion model estimation.''
Sry for it, so I think it might be better that you send an e-mail to the author for this part of code. -
''If I want to re-implement it for online smoothing, do you know how to design it. Give me some ideas, pls.''
In my mind, the Meshflow is a good solution for online-method, or alternatively you can check this repo [https://github.com/liuzhen03/NNDVS]
Hello Ihaippp,Clark.
First of all thanks to both of you for the discussion, it has been very beneficial for me, and also thanks to the author big guy for open sourcing this
After I read your work carefully, I have the following queries in the context of my own research:
I only want to correct my images with the help of gyroscope data and would like to do real time processing, i.e. without borrowing future frames. Since I haven't worked with gyroscopes, I don't really understand how to put gyroscope data to use and would like to get a little bit of information from you. I am sorry if this is a nuisance. Thanks again for your communication and solving some of my doubts.
hi lxc,
what do you mean "I only want to correct my images with the help of gyroscope data", image alignment or RS removal?
"I don't really understand how to put gyroscope data", I think this repo could somehow be exactly what you need
hi lhaippp,
Thank you very much for your quick reply and advice. I'd like to use the gyroscope data to estimate my camera motion and use that to correct my image
hi,
ok; so "I think this repo could somehow be exactly what you need", I recommend you to try it
hi，
Thank you so much for the advice, I'm going to try it now!
How can I save the entire video instead of a gif image？
https://github.com/lhaippp/gyro-video-stabilization/assets/75515605/101d44ff-a0ea-4b8e-982c-fe33c497d6e9
This is the result I saved. Could you please help me check if it is correct.thanks！
hi lhaippp,
I'm sorry to bother you again. After reading your code, I realized that something bad happened after code processing:
I noticed a truncation of the straight lines in the graph after processing
Hi, @lxc5720
I agree with the two points mentioned by @lhaippp , and I would like to add some contents to the second point,
You can find the rolling shutter correction based on quaternion slerp in this link , and I have previously written a series of analytical articles about video stabilization using gyroflow-rust on Zhihu. I hope this could be helpful to you.
Hi, @lhaippp @ClarkWang1214
Thank you very much for your quick reply, I will try the suggestions you have given!
Hi,
I apologize that the weekend has caused me to wait until now to thank you for your replies. I have just tried what you have suggested and it does indeed solve the problem I mentioned before. My next plan is to complete the stabilization of the video along the lines of rolling shutter correction, my idea is as follows: take the 3 angles of the first frame of the video frame as a baseline, and each subsequent frame is a relative angle based on the first frame, and then low-pass filter the relative angles of each frame as a way to achieve stabilization. Is this idea feasible please?
Thanks again for your @ClarkWang1214 @lhaippp answers, they are very helpful!