RecRecNet: Rectangling Rectified Wide-Angle Images by Thin-Plate Spline Model and DoF-based Curriculum Learning
This is the official implementation for RecRecNet (ICCV2023).
Kang Liao†, Lang Nie†, Chunyu Lin, Zishuo Zheng, Yao Zhao
Given a rectified wide-angle image, RecRecNet aims to construct a win-win representation on both image content and boundary, with the perspective of the rectangling technique.
- First win-win representation of the large field-of-view (FoV) vision
- A thin-plate spline (TPS) motion module is proposed to flexibly formulate the non-linear and non-rigid rectangling transformation
- A DoF-based curriculum learning is designed to grasp the progressive deformation rules and relieve the burden of complex structure approximation
- An in-depth analysis of why the deformed image boundary can significantly influence the vision perception models
Using the virtual environment (conda) to run the code is recommended.
conda create -n recrecnet python=3.6
conda activate recrecnet
pip install -r requirements.txt
We constructed the first dataset for the rectified wide-angle rectangling task. The structure of the original rectified wide-angle image was first optimized by an energy function with line-preserving mesh deformation, as proposed in He et al.. And then we carefully filtered all results and repeated the selection process three times. The dataset can be downloaded here: train.zip, test.zip.
Download the pretrained model here and put it into the .\checkpoint
folder.
Generate the curriculum to grasp the progressive deformation rules of rectangling. The source image can be collected from ImageNet or COCO. Please set the suitable
sh scripts/curriculum_gen.sh
Customize the paths of 4-dof dataset, 8-dof dataset, and wide-angle image rectangling dataset, and run:
sh scripts/train.sh
Customize the paths of checkpoint and test set, and run:
sh scripts/test.sh
The rectangling image and its corresponding warping mesh (formed by predicted TPS control points) can be found in the .\results
folder.
If you feel RecRecNet is helpful in your research, please consider referring to it:
@article{liao2023recrecnet,
title={RecRecNet: Rectangling rectified wide-angle images by thin-plate spline model and DoF-based curriculum learning},
author={Liao, Kang and Nie, Lang and Lin, Chunyu and Zheng, Zishuo and Zhao, Yao},
journal={arXiv preprint arXiv:2301.01661},
year={2023}
}