/CSCV

Primary LanguagePythonBSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

ScaleRAFT

code for ScaleRAFT: Cross-Scale Recurrent All-Pairs Field Transforms for 3D Motion Estimation.

(A Robust Method for Extracting 3D Motion from Videos)

Requirements

The code has been tested with PyTorch 1.11.0 and Cuda 11.3.

conda create -n cscv python=3.9
conda activate cscv
pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113

Also need to install via pip

pip install matplotlib==3.5
pip install opencv-python
pip install tqdm
pip install pypng
pip install scipy
pip install einops
pip install tensorboard

Demo

First, download the weights (Demo.pth) from https://drive.google.com/drive/folders/129lbJWkcMwxispcRVXOvUGF12GuHbhX3?usp=drive_link and place it in the checkpoints path.

You need to specify the image path and output path in the Demo.py file (line 92,93,94)

        path1 = '/home/lh/CSCV/00026.jpg'
        path2 = '/home/lh/CSCV/00027.jpg'
        outpath = '/home/lh/CSCV/output'
CUDA_VISIBLE_DEVICES=0 python Demo.py --model=/home/lh/CSCV/checkpoints/Demo.pth --mixed_precision --start=0

soapbox:

soapbox.mp4

parkour:

parkour.mp4

motorbike:

motorbike.mp4

motocross-jump:

motocross-jump.mp4

lady-running:

lady-running.mp4

car-shadow:

car-shadow.mp4

breakdance-flare:

breakdance-flare.mp4

Dog:

00007.mp4

Dataset Configuration

To evaluate/train CSCV, you will need to download the required datasets.

We recommend manually specifying the path in dataset_exp_orin.py , like in line 477 def __init__(self, aug_params=None, split='kitti_test', root='/new_data/datasets/KITTI/',get_depth=0): , '/new_data/datasets/KITTI/' is where you put the KITTI dataset.

You can create symbolic links to wherever the datasets were downloaded in the datasets folder

├── datasets
    ├── KITTI
        ├── testing
        ├── training
        ├── devkit
    ├── FlyingThings3D
        ├── frames_cleanpass
        ├── frames_finalpass
        ├── optical_flow

Pretrained weights for KITTI

Download and place in the checkpoints directory

Train on KITTI

CUDA_VISIBLE_DEVICES=0 python train.py --name raft-cscv --stage kitti --validation kitti --restore_ckpt ../CSCV/checkpotins/cscv_kittitest_6.12.pth --gpus 0 --num_steps 60000 --batch_size 2 --lr 0.000125 --image_size 320 960 --wdecay 0.0001 --gamma=0.85

Test on KITTI

Reproduce the results of Table 1 in the paper

CUDA_VISIBLE_DEVICES=0 python dc_flow_eval.py --model=../CSCV/checkpotins/cscv_kitti_42.08.pth --mixed_precision --start=0