CLIP-Mesh
Official implementation of CLIP-Mesh: Generating textured meshes from text using pretrained image-text models
Nasir Mohammad Khalid,
Tianhao Xie,
Eugene Belilovsky,
Tiberiu Popa
SIGGRAPH ASIA 2022 | arXiv | Project page
Quickstart
(Work in Progress!) - The following Google Collab notebooks are the best way to get started:
 |
 |
|---|---|
| Text to Mesh | Multi Mesh Generation |
 |
 |
|---|---|
| Stylizing a Mesh | Apply CLIP-Mesh to Human Models |
Setup
Clone this repository recursively to get all submodules - use submodule update to get downstream submodules
git clone --recurse-submodules https://github.com/NasirKhalid24/CLIP-Mesh.git
cd CLIP-Mesh
git submodule update --init --recursive
Setup Conda environment and install packages
conda create -n clip-mesh python=3.7
conda activate clip-mesh
conda install pytorch==1.11.0 torchvision==0.12.0 cudatoolkit=10.2 -c pytorch
pip install -r requirements.txt
Install loop subdivison code and DALLE-2 (not that DALLE-2 is from an earlier commit so existing install may not work)
<!-- Install Loop Subdivision -->
cd loop_limitation
pip install .
cd ..
<!-- Install DALLE2 - Diffusion Prior -->
cd DALLE2-pytorch
pip install .
cd ..
<!-- Get DALLE2 Prior Weights -->
mkdir weights
wget https://huggingface.co/spaces/NasirKhalid24/Dalle2-Diffusion-Prior/resolve/main/larger-model.pth -O ./weights/model.pth
Usage
This repo comes with some configs that are passed to main.py using the --config flag
Any of the config paramaters can be overriden by passing them to as arguments to the main.py file so you can have a base .yml file with all your parameters and just update the text prompt to generate something new
An example would be using the given config file for single mesh generation single.yml
# Use all parameters in file
python main.py --config configs/single.yml
# Use all parameters in file but change text prompt
python main.py --config configs/single.yml --text_prompt "a hamburger"
# Use all parameters in file but change text prompt, batch, texture resolution
python main.py \
--config configs/single.yml \
--text_prompt "a hamburger" \
--batch_size 5 \
--texture_resolution 1024
Tips, Tricks, FAQs etc
I recommend checking out the following document as it could answer any doubts that come up (will be updated regularly) - if you still have questions reach out @Nymarius_ or open an issue