Developed at the ENS Paris-Saclay, Centre Borelli and accepted at the CVPR EarthVision Workshop 2022.
Sat-NeRF: Learning Multi-View Satellite Photogrammetry With Transient Objects and Shadow Modeling Using RPC Cameras
Roger Marí, Gabriele Facciolo, Thibaud Ehret
Abstract: We introduce the Satellite Neural Radiance Field (Sat-NeRF), a new end-to-end model for learning multi-view satellite photogrammetry in the wild. Sat-NeRF combines some of the latest trends in neural rendering with native satellite camera models, represented by rational polynomial coefficient (RPC) functions. The proposed method renders new views and infers surface models of similar quality to those obtained with traditional state-of-the-art stereo pipelines. Multi-date images exhibit significant changes in appearance, mainly due to varying shadows and transient objects (cars, vegetation). Robustness to these challenges is achieved by a shadow-aware irradiance model and uncertainty weighting to deal with transient phenomena that cannot be explained by the position of the sun. We evaluate Sat-NeRF using WorldView-3 images from different locations and stress the advantages of applying a bundle adjustment to the satellite camera models prior to training. This boosts the network performance and can optionally be used to extract additional cues for depth supervision.
If you find this code or work helpful, please cite:
@inproceedings{mari2022sat,
title={{Sat-NeRF}: Learning Multi-View Satellite Photogrammetry With Transient Objects and Shadow Modeling Using {RPC} Cameras},
author={Mar{\'\i}, Roger and Facciolo, Gabriele and Ehret, Thibaud},
booktitle={2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
pages={1310-1320},
year={2022}
}
- This project works with multiple conda environments, named
satnerf,s2pandba.
satnerfis the only strictly necessary environment. It is required to train/test SatNeRF.s2pis used to additionally evaluate a satellite MVS pipeline relying on classic computer vision methods.bais used to bundle adjust the RPCs of the DFC2019 data.
To create the conda environments you can use the setup scripts, e.g.
conda init && bash -i setup_satnerf_env.sh
Warning: If some libraries are not found, it may be necessary to update the environment variable LD_LIBRARY_PATH before launching the code:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib
where $CONDA_PREFIX is the path to your conda or miniconda environment (e.g. /mnt/cdisk/roger/miniconda3/envs/satnerf).
You can download here the training and test datasets, as well as some pretrained models.
Example command to generate a surface model with Sat-NeRF:
(satnerf) $ export dataset_dir=/mnt/cdisk/roger/EV2022_satnerf/dataset
(satnerf) $ export pretrained_models=/mnt/cdisk/roger/EV2022_satnerf/pretrained_models
(satnerf) $ python3 create_satnerf_dsm.py Sat-NeRF $pretrained_models/JAX_068 out_dsm_path/JAX_068 28 $pretrained_models/JAX_068 $dataset_dir/root_dir/crops_rpcs_ba_v2/JAX_068 $dataset_dir/DFC2019/Track3-RGB-crops/JAX_068 $dataset_dir/DFC2019/Track3-TruthExample command for novel view synthesis with Sat-NeRF:
(satnerf) $ python3 eval_satnerf.py Sat-NeRF $pretrained_models/JAX_068 out_eval_path/JAX_068 28 val $pretrained_models/JAX_068 $dataset_dir/root_dir/crops_rpcs_ba_v2/JAX_068 $dataset_dir/DFC2019/Track3-RGB-crops/JAX_068 $dataset_dir/DFC2019/Track3-TruthExample command:
(satnerf) $ python3 main.py --model sat-nerf --exp_name JAX_068_ds1_sat-nerf --root_dir /mnt/cdisk/roger/Datasets/SatNeRF/root_dir/crops_rpcs_ba_v2/JAX_068 --img_dir /mnt/cdisk/roger/Datasets/DFC2019/Track3-RGB-crops/JAX_068 --cache_dir /mnt/cdisk/roger/Datasets/SatNeRF/cache_dir/crops_rpcs_ba_v2/JAX_068_ds1 --gt_dir /mnt/cdisk/roger/Datasets/DFC2019/Track3-Truth --logs_dir /mnt/cdisk/roger/Datasets/SatNeRF_output/logs --ckpts_dir /mnt/cdisk/roger/Datasets/SatNeRF_output/ckptsThe create_satellite_dataset.py script can be used to generate input datasets for SatNeRF from the open-source DFC2019 data. The Track3-RGB and Track3-Truth folders are needed.
We encourage you to use the bundle_adjust package, available here, to ensure your dataset employs highly accurate RPC camera models. This will also allow aggregating depth supervision to the training and consequently boost the performance of the NeRF model.
(ba) $ python3 create_satellite_dataset.py JAX_068 $dataset_dir/DFC2019 out_dataset_path/JAX_068Alternatively, if you prefer not installing bundle_adjust, it is also possible to use the flag --noba to create the dataset using the original RPC camera models from the DFC2019 data.
(ba) $ python3 create_satellite_dataset.py JAX_068 $dataset_dir/DFC2019 out_dataset_path/JAX_068 --nobaThe --splits flag can also be used to generate the train.txt and test.txt files.
The script study_depth_supervision.py produces an interpolated DSM with the initial depths given by the 3D keypoints output by bundle_adjust.
Example command:
(satnerf) $ python3 study_depth_supervision.py Sat-NeRF+DS $pretrained_models/JAX_068 out_DS_study_path/JAX_068 $dataset_dir/root_dir/crops_rpcs_ba_v2/JAX_068 $dataset_dir/DFC2019/Track3-RGB-crops $dataset_dir/DFC2019/Track3-TruthThe script study_solar_interpolation.py can be used to visualize images of the same AOI rendered with different solar direction vectors.
Example command:
(satnerf) $ python3 study_solar_interpolation.py Sat-NeRF $pretrained_models/JAX_068 out_solar_study_path/JAX_068 28 $pretrained_models/JAX_068 $dataset_dir/root_dir/crops_rpcs_ba_v2/JAX_068 $dataset_dir/DFC2019/Track3-RGB-crops/JAX_068 $dataset_dir/DFC2019/Track3-TruthWe compare the DSMs learned by SatNeRF with the equivalent DSMs obtained from manually selected multiple stereo pairs, reconstructed using the S2P pipeline.
More details of the classic satellite MVS reconstruction process can be found here.
Use the script eval_s2p.py to reconstruct an AOI using this methodology.
(s2p) $ python3 eval_s2p.py JAX_068 /mnt/cdisk/roger/Datasets/SatNeRF/root_dir/fullaoi_rpcs_ba_v1/JAX_068 /mnt/cdisk/roger/Datasets/DFC2019 /mnt/cdisk/roger/nerf_output-crops3/results --n_pairs 10