[ECCV24] LayeredFlow Benchmark

We introduce LayeredFlow, a real world benchmark with multi-layer annotations for optical flow of non-Lambertian objects, featuring 150k optical flow and stereo pairs across 185 diverse scenes with 360 unique objects.

LayeredFlow enables a new task called multi-layer optical flow. We also offer a large-scale, densely-annotated synthetic dataset with 60k images from 30 scenes for training. Fine-tuning on this dataset enhances the performance of existing optical flow methods on non-Lambertian objects.

If you find LayeredFlow useful for your work, please consider citing our academic paper:

LayeredFlow: A Real-World Benchmark for Non-Lambertian Multi-Layer Optical Flow

Hongyu Wen, Erich Liang, Jia Deng

@article{wen2024layeredflow,
    title={LayeredFlow: A Real-World Benchmark for Non-Lambertian Multi-Layer Optical Flow}, 
    author={Hongyu Wen and Erich Liang and Jia Deng},
    journal={arXiv preprint arXiv:2409.05688},
    year={2024},    
}

Installation

conda env create -f env.yaml
conda activate layeredflow

LayeredFlow Benchmark

Download

Download the validation set (images + ground-truth) and test set (images) here.

Unzip the data. Make soft links to the data under RAFT/datasets and MultiRAFT/datasets folder.

├── datasets
    ├── layeredflow
        ├── test
        ├── val

Evaluation on Validation Set

You can find the pre-trained checkpoints here.

To evaluate RAFT on single layer subset of the benchmark using pre-trained checkpoints, download corresponding checkpoints and put them in RAFT/checkpoints, run

cd RAFT
python3 evaluate_firstlayer.py --checkpoint first_layer_S+L.pth --dataset layeredflow # first layer
python3 evaluate_lastlayer.py --checkpoint last_layer_L.pth --dataset layeredflow # last layer

To evaluate MultiRAFT on full set of the benchmark using pre-trained checkpoints, download corresponding checkpoints and put them in MultiRAFT/checkpoints, run

cd MultiRAFT
python3 evaluate.py --checkpoint MultiRAFT_S+L.pth --dataset layeredflow --mixed_precision

Evaluation on Test Set

To evaluate your model on the test set and compare your results with the baseline, you need to submit your flow predictions to the evaluation server.

Navigate to the RAFT directory and execute the following command to create your submission. A folder named "layeredflow_submission" will be created.

cd RAFT
python3 evaluate_firstlayer.py --checkpoint your_checkpoint --dataset layeredflow --create_submission # first_layer benchmark
python3 evaluate_lastlayer.py --checkpoint your_checkpoint --dataset layeredflow --create_submission # last_layer benchmark

cd MultiRAFT
python3 evaluate.py --checkpoint your_checkpoint --dataset layeredflow --create_submission # multi_layer benchmark

Submit your predictions to the evaluation server using the command below. Make sure to replace placeholders with your actual email, submission path, and method name:

python3 upload_submission.py --email your_email --path path_to_your_submission --method_name your_method_name --benchmark first_layer

Upon submission, you will receive a unique submission ID, which serves as the identifier for your submission. Results are typically emailed within 1 hour. Please note that each email user may upload only three submissions every seven days.

To make your submission public, run the command below. Please replace the placeholders with your specific details, including your submission ID, email, and method name. You may specify the publication name, or use "Anonymous" if the publication is under submission. It's optional to provide URLs for the publication and code.

python3 modify_submission.py --id submission_id --email your_email --anonymous False --method_name your_method_name --publication "your publication name" --url_publication "https://your_publication" --url_code "https://your_code"

Synthetic Data Generation

Pre-rendered Dataset

You can find the pre-rendered images here.

To unzip the dataset, download all the files then run:

cat *.tar.gz.* | tar xvfz -

Set up Blender

To generate multiple layer ground truth from blender, you will need to build customized blender.

First, clone the Blender source code to your machine. Using version 3.6 (v3.6.14 is tested for MacOS).

git clone --depth 1 --branch v3.6.14 git@github.com:blender/blender.git

Next, apply the multi-layer ground truth patch.

git apply multiple-layer-ground-truth.patch

Then build blender from source following instructions.

Rendering Scripts

Coming Soon!

Acknowledgements

This project relies on code from existing repositories: RAFT and ptlflow. We thank the original authors for their excellent work.

princeton-vl/LayeredFlow