This is the official code repository for the paper "Generalizable Pose Estimation Using Implicit Scene Representations" presented at ICRA 2023.
![](https://private-user-images.githubusercontent.com/10887480/308959862-6ded7ac0-0829-484f-94f8-b0e281d0669a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ4MDE2NDAsIm5iZiI6MTcyNDgwMTM0MCwicGF0aCI6Ii8xMDg4NzQ4MC8zMDg5NTk4NjItNmRlZDdhYzAtMDgyOS00ODRmLTk0ZjgtYjBlMjgxZDA2NjlhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI3VDIzMjkwMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTZiMGM3Y2EyMjdkYmRlYjhiYzI0MjBhYjUwMjJiNTg4OGY2MWUwZDRiNDg4ZTFiYjg5YTY1ODM0MTAxZjg3NjQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.L9s_a_fVGRhFWfkk5zv3R1gaH3beKlrtPDjWVwUWexU)
![](https://private-user-images.githubusercontent.com/10887480/308959815-d383d48e-4ff1-4ae6-ae55-ae61b06da008.gif?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ4MDE2NDAsIm5iZiI6MTcyNDgwMTM0MCwicGF0aCI6Ii8xMDg4NzQ4MC8zMDg5NTk4MTUtZDM4M2Q0OGUtNGZmMS00YWU2LWFlNTUtYWU2MWIwNmRhMDA4LmdpZj9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI3VDIzMjkwMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTdiOTBmODg3NjA3NWRkNGNlYjBhNWFhMzkyMWUxZjg3MzU1ZGZjNDkxYzAzYjQwNTY3MmM1MGFhMDg2OGJmNTMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.6YJEHSsLo24NfeG9d1SdcmCoZD1Xg3nh9zOw7332fiY)
Please see our project website for demo videos and more details about our method!
The code for the σSRN model is based on the original SRN codebase. The modified code is located in the sigma_srn folder.
git clone --recurse-submodules https://github.com/vaibhavsaxena11/inverse-sigma-srn.git
conda env create -f environment.yml
conda activate sigma-srns
Download Shapenet v2 cars and chairs classes dataset from here. Fetch the cars_train.zip, cars_train_val.zip, chairs_train.zip and chairs_train_val.zip files.
See python train.py --help
for all train options.
Example train call:
python train.py --data_root [path to directory with dataset] \
--logging_root [path to directory where tensorboard summaries and checkpoints should be written to] \
--no_validation \
--img_sidelengths 64 \
--batch_size_per_img_sidelength 10
To monitor progress, the training code writes tensorboard summaries every 100 steps into a "events" subdirectory in the logging_root.
See python pose.py --help
for all pose evaluation options.
Example pose call:
python pose.py --train_data_root [path to directory with dataset] \
--val_data_root [path to directory with train_val dataset - unseen poses for evaluation] \
--logging_root [path to directory where the results are stored] \
--checkpoint_path [path to the σSRN model checkpoint *.pth file]
--loss l1
For each image in the validation set (from an unseen camera pose), pose.py will optimize the camera pose parameters to move the view to match the input image. Multiple starting poses are used, and the pose with the lowest loss is chosen as the estimated camera pose associated with the input image. For each pose optimization, the rendered views as the camera parameters move towards the target pose are stored in the logging directory. If plotting is enabled (on by default), error plots are stored in the logging directory as well. For each input target image, a pose.txt file indicates the best estimate.
If you find this repository useful, please cite our work:
@misc{saxena2023generalizable,
title={Generalizable Pose Estimation Using Implicit Scene Representations},
author={Vaibhav Saxena and Kamal Rahimi Malekshan and Linh Tran and Yotto Koga},
year={2023},
eprint={2305.17252},
archivePrefix={arXiv},
primaryClass={cs.CV}
}