Spec-NeRF: Multi-Spectral Neural Radiance Fields
Jiabao Li, Yuqi Li*, Ciliang Sun, Chong Wang, and Jinhui Xiang
Intro
Spec-NeRF jointly optimizes the degradation parameters and achieves high-quality multi-spectral image reconstruction results at novel views, which only requires a low-cost camera (like a phone camera but in RAW mode) and several off-the-shelf color filters. We also provide real scenarios and synthetic datasets for related studies.
Video Demo
With recovered spectral information of the scenario, we can achieve several art effects, like
Change the filters
filter.mp4
Change the camera's SSF
ssf.mp4
Change the ambient light source spectrum
ambient.mp4
Preliminaries
We conduct our experiments based on TensoRF, please use the branch named public in our repository and feel free to report issues, we'd really appreciate it!
Tested on Ubuntu 18 / Windows 11 + Pytorch 1.11 + cuda 11.3
Dataset
Download the two types of datasets (real senario and synthetic one) from google drive.
For the real dataset, please first separate the images into pose-based groups since all raw images are in the folder RAW by running python split_allimg2filterfixed_classify.py --scene_dir <your full path>/multi-view-MSI/filter19/xjhdesk --filter_num 20 --legacy 0 --angle_num 9 --img_ext tiff as administrator or root.
This will generate nine folders named pose?img, each contains the images filtered by all color filters, which are the symbolic links point to the ones in RAW folder, so remember do not delete the RAW folder.
Quick Start
Check what's the config file in start/start.bat file and execute ./start | start.bat
or try
python train.py --config ./configs/<your config file>.txt