Regarding the issue of missed frames

Question

Regarding the issue of missed frames

Closed this issue 8 months ago · 7 comments

Hi,I have a video with a frame rate of 30 frames per second, but it experiences frame dropping. Assuming it drops one frame every 30 frames, should I set FS to 29 or FS to 30 in the training config?

408550969 commented 8 months ago

Thanks!

Answer 1 · 2024-04-16T06:38:30.000Z

You should be fine with setting FS to 30. 29 versus 30 is a fairly minor discrepancy and shouldn't affect your results significantly. If you want, you can try looking into video interpolation methods to try and correct the frame dropping, maybe something like this using ffmpeg. Again, I should stress for the task of rPPG you really shouldn't worry about this small of a discrepancy.

Answer 2 · 2024-04-16T08:40:17.000Z

Thanks, I have another question. Will the delay between the image and label cause the model to fail to converge? I collected some videos and found that when using EfficientPhys, as long as the difference between the image (already calculated camera delay) and the label exceeds plus or minus 133ms, the model cannot converge. Is RPPG very sensitive to latency?

Answer 3 · 2024-04-18T02:57:18.000Z

Hi @408550969,

There's definitely a possibility that some combinations of models and datasets are more sensitive to synchronization error, which in turn could lead to a failure to converge. Can you share more details (e.g., how you identified the model being unable to converge, such as plots of the training and loss curves which can be produced by this toolbox)?

Usually this sensitivity has more to do with the loss function than a specific model from what I understand. If correcting the synchronization error is challenging, you could try using loss functions such as the Maximum Cross-Correlation (MCC) as suggested by this paper.

Answer 4 · 2024-04-23T08:25:07.000Z

I identified by testing the MAE of both the test set and the training set that when the delay is large, not only does the MAE of the test set reach tens, but the MAE of the training set also reaches tens.
Thanks, I will consider using MCC as the loss.

Answer 5 · 2024-04-24T10:10:22.000Z

I have another question, does video encoding have a significant impact on accuracy? For example, if I use lossy compression, what is the difference in MAE compared to lossless images?

Answer 6 · 2024-04-28T20:52:54.000Z

Hi @408550969,

I recommend reading section 10.5 (titled 'Video Compression') of this excellent review article, as well as any of the cited works in that section that sound interesting to you and relate to the effects of video compression on the task of rPPG. To put it briefly and based on my understanding, yes, there is a difference and I'd expect compression that has more temporal effects to subsequently have a greater effect on making your SNR and possibly your MAE worse.