/optical_flow_playground

Educational Project for ML Research Winter School by UCU

Primary LanguagePython

Optical Flow Overview (educational project)

Description

The main goal of this repo is to learn the Optical Flow task. It includes:

  • task overview
  • research of existing approaches
  • results of reprodusing
  • practice in Attacking Optical Flow

Optical Flow - an Overview Notes

Optical Flow Estimation is the problem of finding pixel-wise motions between consecutive images.

Sparse optical flow of traffic Optical flow problem
drawing drawing

[ images source, paperswithcode page ]

Tested approaches

Attacking Optical Flow

Implementation of a patch with noise or other content into frames can break the process of optical flow estimation.

This approach includes patches with a default value or the content of the patch is determined by the optimization problem. This customizes the patch and improves the results.

Example of attacks from Attacking Optical Flow paper:

drawing

Results

GMA model:

Unattacked Frames Attacked Frames Unattacked Flow Attacked Flow
drawing drawing drawing drawing
RAFT model
Unattacked Frames Attacked Frames Unattacked Flow Attacked Flow
drawing drawing drawing drawing
FlowNet2 model
Unattacked Frames Attacked Frames Unattacked Flow Attacked Flow
drawing drawing drawing drawing

Metric: Mean Squared Error (MSE)

Model MSE Noise Type
FlowNet2 56.9827 Patch
GMA 2.5502 Patch

Conclusions

For more effective patch attacks, more complex approaches are needed. Gaussian noise and even patches optimized for testing networks will not add a significant amount of noise.