/Anything-3D

Segment-Anything + 3D. Let's lift the anything to 3D.

Primary LanguagePython

logo

🎉🎉🎉Welcome to the Anything-3D GitHub repository!🎉🎉🎉

Here we present a project where we combine Segment Anything with a series of 3D models to create a very interesting demo. This is currently a small project, but we plan to continue improving it and creating more exciting demos.

Contributions are highly Welcomed!🤝🙌

🤩 Anything-3D-Objects

In this section, we showcase the results of combining Segment Anything with 3DFuse to segment and reconstruct 3D objects in the wild. Check out the following table for segmentation results and the corresponding 3D object:

Segmentation Result
1
2

🔥 Anything-3DNovel-View

In this section, we demonstrate the combination of Segment Anything with Zero 1-to-3 to generate novel views of 3D objects. Check out the following images:

1 2 3

🥳 Anything-NeRF

In this section, we showcase the integration of Segment Anything with NeRF to generate new perspectives of objects set against intricate backgrounds. When an object is positioned in front of a plain, perspective-less background, NeRF typically struggles to reconstruct the scene. However, by eliminating the background, we can enhance NeRF's performance and facilitate more accurate reconstructions of scenes with objects presented in novel views.

Segmentation-1 Segmentation-2 Result
1

😎 Any-3DFace

In this section, we showcase the results of combining Segment Anything with HRN for accurate and detailed face reconstruction from in-the-wild images. Check out the following table for segmentation results and the corresponding face reconstruction:

Segmentation Result
1
3
3

💘 Acknowledgements

We would like to acknowledge the following projects for their valuable contributions:

Citation

If you find this project helpful for your research, please consider citing the following BibTeX entry.

@misc{shen2023anything3d,
    title={Anything-3D: Towards Single-view Anything Reconstruction in the Wild}, 
    author={Qiuhong Shen and Xingyi Yang and Xinchao Wang},
    year={2023},
    eprint={2304.10261},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

And other projects

@article{kirillov2023segany,
    title={Segment Anything}, 
    author={Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C. and Lo, Wan-Yen and Doll{\'a}r, Piotr and Girshick, Ross},
    journal={arXiv:2304.02643},
    year={2023}
}
@misc{liu2023zero1to3,
    title={Zero-1-to-3: Zero-shot One Image to 3D Object}, 
    author={Ruoshi Liu and Rundi Wu and Basile Van Hoorick and Pavel Tokmakov and Sergey Zakharov and Carl Vondrick},
    year={2023},
    eprint={2303.11328},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}
@inproceedings{Lei2023AHR,
    title={A Hierarchical Representation Network for Accurate and Detailed Face Reconstruction from In-The-Wild Images},
    author={Biwen Lei and Jianqiang Ren and Mengyang Feng and Miaomiao Cui and Xuansong Xie},
    year={2023}
}
@article{seo2023let,
    title={Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation},
    author={Seo, Junyoung and Jang, Wooseok and Kwak, Min-Seop and Ko, Jaehoon and Kim, Hyeonsu and Kim, Junho and Kim, Jin-Hwa and Lee, Jiyoung and Kim, Seungryong},
    journal={arXiv preprint arXiv:2303.07937},
    year={2023}
}
@inproceedings{mildenhall2020nerf,
    title={NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis},
    author={Ben Mildenhall and Pratul P. Srinivasan and Matthew Tancik and Jonathan T. Barron and Ravi Ramamoorthi and Ren Ng},
    year={2020},
    booktitle={ECCV},
}