Training Script

[santurini]

Hello,
I would like to train my own optical flow using this scheme, what changes should I do ninth RealFlow.py script?
Reading the paper I understood that only the Optical Flow gets trained while DPT and BHF are fixed.
This means I should remove torch.no_grad() for the flow_estimation part but which loss should I use to backpropagate? Computed between which pair of tensors?

Really look forward to know better this work, I really like it!

[coolbeam]

Hi,
To train your own optical flow model using the approach in this paper, you can follow these steps:

1. Use the provided code (RealFlow.py or demo.py) to generate optical flow data from your own video data using pre-trained optical flow model and DPT model.
2. Once you have the generated optical flow dataset, fine-tune the optical flow network or train any other optical flow model with its corresponding training code (e.g., RAFT, GMA).
3. Once you have a better optical flow model, you can generate improved datasets using the trained model. So, we can repeat the process of dataset generation and fine-tuning the optical flow model multiple times to achieve better results.

Please note that we don't provide the training code for optical flow models. You can refer to the training code of popular models like RAFT or GMA to train your own optical flow models and just modify a dataloader code for reading your generated data into the training scripts.

[santurini]

Thank you very much!
This means I can generate a huge dataset simply by stacking together multiple videos and pass the in the form of (frame1, frame2) right?

But shouldn't be the performance be limited by the goodness of the model? I mean if I train my model on data generated by the model itself how should this help in fine-tuning the same model?