dreamfields-3D

A colab friendly toolkit to generate 3D mesh model / video / nerf instance / multiview images of colourful 3D objects by text and image prompts input. Edited by Shengyu Meng (Simon)

Check the colab notebook for usage.

Dreamfields-3D is modified from dreamfields-torch and dreamfields, please check the Credits.md for details.

Preview.mp4

Example generated by text prompt: "a beautiful painting of a flower tree, by Chiho Aoshima, Long shot, surreal"

Main Contributions:

Integrating Video generation.
Export obj & ply model with vertex colour.
Export 360° Video of final model.
Visualizing the training progress and preview the output video in colab.
Improve the generation quality.
- Implements multiple CLIP model.
- Improve the pre-process of the renderings before feeding into CLIP.
Add more useful arguments.
Organize the colab notebook.

Future update plan:

Use different CLIP models simultaneously.
Apply the image prompt only in certain assigned direction.
Read existing mesh as NeRF instance then modify by text / image prompts.
Reduce the usage of GPU RAM occupation in training.

I am a coding beginner, cannot promise for the progress, and welcome to contribute to this repository.

Compatibility:

About system:
- Colab: Pass on google Colab (tested on A100/v100/P100 GPU at 08/09/2022)
- Ubuntu: The previous version (dreamfields-torch) has successfully ran on Ubuntu 18.04 with RTX 3090. Not tested for the dreamfields-3D yet, but mostly should be fined.
- Windows: It should be work in windows with proper environment, but I failed to build the raymarching extension in different windows machine. More test will be required.
- For locally running, please refer to the installation guide of dreamfields-torch bellow.
About Exporting 3D model:
- The 3D models will be exported as obj and ply format with vertex colour.
- Mesh with vertex colour could be seen in meshLab and Rhino3D.
- For viewing colour in Blender, please import the ply model frist, then create a new material, and plug a Color Attribute node into the base color in shader editor, then you should see the vertex colour.
About GUI:
- GUI is supported only when running locally. However, some new features maybe not available in GUI model. I would recommend to train without GUI first, then run test on the pretrained model in GUI, to browse the final 3D NeRF instance.

👇 Bellow readme from the dreamfields-torch repository.

dreamfields-torch (WIP)

A pytorch implementation of dreamfields as described in Zero-Shot Text-Guided Object Generation with Dream Fields.

An example of a generated neural field by prompt "cthulhu" viewed in real-time:

cthulhu.mp4

Install

The code framework is based on torch-ngp.

git clone https://github.com/ashawkey/dreamfields-torch.git
cd dreamfields-torch

Install with pip

pip install -r requirements.txt

install customized verion of pymarchingcubes

bash scripts/install_PyMarchingCubes.sh

Build extension (optional)

By default, we use load to build the extension at runtime. However, this may be inconvenient sometimes. Therefore, we also provide the setup.py to build each extension:

# install all extension modules
bash scripts/install_ext.sh
# if you want to install manually, here is an example:
cd raymarching
python setup.py build_ext --inplace # build ext only, do not install (only can be used in the parent directory)
pip install . # install to python path (you still need the raymarching/ folder, since this only install the built extension.)

Tested environments

Ubuntu 20 with torch 1.10 & CUDA 11.3 on a TITAN RTX.
Windows 10 with torch 1.11 & CUDA 11.3 on a RTX 3070.

Currently, --ff only supports GPUs with CUDA architecture >= 70. For GPUs with lower architecture, --tcnn can still be used, but the speed will be slower compared to more recent GPUs.

Usage

First time running will take some time to compile the CUDA extensions.

# text-guided generation
python main_nerf.py --text "cthulhu" --workspace trial --cuda_ray --fp16

# use the GUI
python main_nerf.py --text "cthulhu" --workspace trial --cuda_ray --fp16 --gui

# [experimental] image-guided generation (also use the CLIP loss)
python main_nerf.py --image /path/to/image --workspace trial --cuda_ray --fp16

check the scripts directory for more provided examples.

Difference from the original implementation

Mip-nerf is not implemented, currently only the original nerf is supported.
Sampling poses with an elevation range in [-30, 30] degrees, instead of fixed at 30 degree.
Use the origin loss.

Update Logs

5.18: major update.
3.16: basic reproduction.

Acknowledgement

The great paper and official JAX implementation of dreamfields:

@article{jain2021dreamfields,
    author = {Jain, Ajay and Mildenhall, Ben and Barron, Jonathan T. and Abbeel, Pieter and Poole, Ben},
    title = {Zero-Shot Text-Guided Object Generation with Dream Fields},
    journal = {arXiv},
    month = {December},
    year = {2021},
}

The GUI is developed with DearPyGui.

HelixNGC7293/dreamfields-3D