ximinng/PyTorch-SVGRender

SVG Differentiable Rendering: Generating vector graphics using neural networks. Support: text-to-SVG, Image-to-SVG, SVG Editing.

Pytorch-SVGRender

Pytorch-SVGRender: The go-to library for differentiable rendering methods for SVG generation.

Updates • Table of Contents • Installation • Quickstart • FAQ • TODO • Contribution • Acknowledgment • Citation • Licence

🔥 Recent Updates

• [12/2023] 🔥 We open-sourced Pytorch-SVGRender V1.0.

💡 Table of Contents

^{▴ Back to top}

1. Image Vectorization

• DiffVG: Differentiable Vector Graphics Rasterization for Editing and Learning (SIGGRAPH 2020)

  [Project] [Paper] [Code]

  DiffVG is a differentiable rasterizer for 2D vector graphics. This repository is heavily based on DiffVG.

• LIVE: Towards Layer-wise Image Vectorization (CVPR 2022)

  [Project] [Paper] [Code]

• CLIPasso: Semantically-Aware Object Sketching (SIGGRAPH 2022)

  [Project] [Paper] [Code]

• CLIPascene: Scene Sketching with Different Types and Levels of Abstraction (ICCV 2023)

  [Project] [Paper] [Code]

2. Text-to-SVG Synthesis

• CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders (NIPS 2022)

  [Paper] [Code]

• StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis

  [Live] [Paper] [Code]

• CLIPFont: Texture Guided Vector WordArt Generation (BMVC 2022)

  [Paper] [Code]

• VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models (CVPR 2023)

  [Project] [Paper]

• DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models (NIPS 2023)

  [Project] [Live] [Paper] [Code]

• Word-As-Image for Semantic Typography (SIGGRAPH 2023)

  [Project] [Paper] [Code]

• SVGDreamer: Text Guided SVG Generation with Diffusion Model (CVPR 2024)

  [Project] [Paper] [Code] [Blog]

⚙️ Installation

You can follow the steps below to quickly get up and running with PyTorch-SVGRender. These steps will let you run quick inference locally.

In the top level directory run,

sh script/install.sh

Note: Make sure that the script file has execution permissions (you can give them using chmod +x script.sh), and then run the script.

or using docker,

docker run --name svgdreamer --gpus all -it --ipc=host ximingxing/svgrender:v1 /bin/bash

For more information, please refer to the Install.md.

👩‍🎨🎨 Quickstart

^{▴ Back to top}

For more information, read the docs.

1. Basic Usage

DiffVG vectorizes any raster images:

python svg_render.py x=diffvg target='./data/fallingwater.png'
# change 'num_paths' and 'num_iter' for better results
python svg_render.py x=diffvg target='./data/fallingwater.png' x.num_paths=512 x.num_iter=2000

LIVE vectorizes the raster emojis images (in original PNG format):

python svg_render.py x=live target='./data/simile.png'
# change 'num_paths' and 'schedule_each' for better results
python svg_render.py x=live target='./data/simile.png' x.num_paths=5 x.schedule_each=1

CLIPasso synthesizes vectorized sketches from images:

note: first download the U2Net model sh script/download_u2net.sh.

python svg_render.py x=clipasso target='./data/horse.png'

CLIPascene synthesizes vectorized sketches from images:

note: first download the U2Net model sh script/download_u2net.sh.

python svg_render.py x=clipascene target='./data/ballerina.png'

CLIPDraw synthesizes SVGs based on text prompts:

python svg_render.py x=clipdraw "prompt='a photo of a cat'"

StyleCLIPDraw synthesizes SVG based on a text prompt and a reference image:

python svg_render.py x=styleclipdraw "prompt='a photo of a cat'" target='./data/starry.png'

CLIPFont styles vector fonts according to text prompts:

python svg_render.py x=clipfont "prompt='Starry Night by Vincent van gogh'" target='./data/alphabet1.svg'

---

Because the following methods rely on stable diffusion, add diffuser.download=True to the command the first time you run the script.

SVGDreamer generates various styles of SVG based on text prompts. It supports the use of six vector primitives, including Iconography, Sketch, Pixel Art, Low-Poly, Painting, and Ink and Wash.

# primitive: iconography
## 1. German shepherd
python svg_render.py x=svgdreamer "prompt='A colorful German shepherd in vector art. tending on artstation.'" save_step=50 x.guidance.n_particle=6 x.guidance.vsd_n_particle=4 x.guidance.phi_n_particle=2 result_path='./svgdreamer/GermanShepherd'
## 2. sydney opera house
python svg_render.py x=svgdreamer "prompt='Sydney opera house. oil painting. by Van Gogh'" save_step=50 x.guidance.n_particle=6 x.guidance.vsd_n_particle=4 x.guidance.phi_n_particle=2 x.num_paths=512 result_path='./svgdreamer/SydneyOperaHouse'
# primitive: low-ploy
python svg_render.py x=svgdreamer "prompt='A picture of a bald eagle. low-ploy. polygon'" x.style='low-poly' save_step=50 x.guidance.n_particle=6 x.guidance.vsd_n_particle=4 x.guidance.phi_n_particle=2 x.grid=30 x.guidance.num_iter=1000 result_path='./svgdreamer/BaldEagle'
# primitive: pixel-art
python svg_render.py x=svgdreamer "prompt='Darth vader with lightsaber. ultrarealistic. pixelart. trending on artstation.'" x.style='pixelart' save_step=50 x.guidance.n_particle=6 x.guidance.vsd_n_particle=4 x.guidance.phi_n_particle=2 x.guidance.num_iter=1000 result_path='./svgdreamer/DarthVader'
# primitive: painting
python svg_render.py x=svgdreamer "prompt='self portrait of Van Gogh. oil painting. cmyk portrait. multi colored. defiant and beautiful. cmyk. expressive eyes.'" x.style='painting' save_step=50 x.guidance.n_particle=6 x.guidance.vsd_n_particle=4 x.guidance.phi_n_particle=2 x.guidance.t_schedule='randint' x.num_paths=1500 result_path='./svgdreamer/VanGogh_portrait'
# primitive: sketch
python svg_render.py x=svgdreamer "prompt='A free-hand drawing of A speeding Lamborghini. black and white drawing.'" x.style='sketch' save_step=50 x.guidance.n_particle=6 x.guidance.vsd_n_particle=4 x.guidance.phi_n_particle=2 x.guidance.t_schedule='randint' x.num_paths=128 result_path='./svgdreamer/Lamborghini'
# primitive: ink and wash
python svg_render.py x=svgdreamer "prompt='Big Wild Goose Pagoda. ink style. Minimalist abstract art grayscale watercolor.'" x.style='ink' save_step=50 x.guidance.n_particle=6 x.guidance.vsd_n_particle=4 x.guidance.phi_n_particle=2 x.guidance.t_schedule='randint' x.num_paths=128 x.width=6 result_path='./svgdreamer/BigWildGoosePagoda'

VectorFusion synthesizes SVGs in various styles based on text prompts:

# Iconography style
python svg_render.py x=vectorfusion x.style='iconography' "prompt='a panda rowing a boat in a pond. minimal flat 2d vector icon. lineal color. trending on artstation.'"
# PixelArt style
python svg_render.py x=vectorfusion x.style='pixelart' "prompt='a panda rowing a boat in a pond. pixel art. trending on artstation.'"
# Sketch style
python svg_render.py x=vectorfusion x.style='sketch' "prompt='a panda rowing a boat in a pond. minimal 2d line drawing. trending on artstation.'"

Following SVGDreamer, we've added three additional styles (Paining, Ink and Wash and low-ploy) to VectorFusion.

DiffSketcher synthesizes vector sketches based on text prompts:

# DiffSketcher
python svg_render.py x=diffsketcher "prompt='a photo of Sydney opera house'" x.token_ind=5 seed=8019
# DiffSketcher, variable stroke width
python svg_render.py x=diffsketcher "prompt='a photo of Sydney opera house'" x.token_ind=5 x.optim_width=True seed=8019
# DiffSketcher RGBA version
python svg_render.py x=diffsketcher "prompt='a photo of Sydney opera house'" x.token_ind=5 x.optim_width=True x.optim_rgba=True x.optim_opacity=False seed=8019
# DiffSketcher + style transfer
python svg_render.py x=stylediffsketcher "prompt='The French Revolution. highly detailed. 8k. ornate. intricate. cinematic. dehazed. atmospheric. oil painting. by Van Gogh'" x.token_ind=4 x.num_paths=2000 target='./data/starry.png' seed=876809

Word-As-Image follow a text prompt to style a letter in a word:

# Inject the meaning of the word bunny into the 'Y' in the word 'BUNNY'
python svg_render.py x=wordasimage x.word='BUNNY' prompt='BUNNY' x.optim_letter='Y'
# Change font: 'LuckiestGuy-Regular', default: 'KaushanScript-Regular'
python svg_render.py x=wordasimage x.word='DRAGONFLY' prompt='Dragonfly' x.optim_letter='Y' x.font='LuckiestGuy-Regular'

2. SDS Loss based Approach

This is achieved by utilizing a pretrained text-to-image diffusion model as a strong image prior to supervise the training of the PyDiffVG, enabling rendering SVG alignment with the text. This remarkable capability is fundamentally grounded in the use of Score Distillation Sampling (SDS). SDS acts as the core mechanism that lifts raster images from diffusion models to the SVG domain, enabling the training of SVG parameters without images. This includes the methods VectorFusion, DiffSketcher and SVGDreamer.

We only compare the performance of SDS, which means that no other loss is used:

# SDS loss
python svg_render.py x=vectorfusion "prompt='a panda rowing a boat in a pond. minimal flat 2d vector icon. lineal color. trending on artstation.'"
# Input Augmentation SDS loss (LSDS loss)
python svg_render.py x=vectorfusion x.style='sketch' "prompt='an elephant. minimal 2d line drawing. trending on artstation.'" x.skip_live=True
# Input Augmentation SDS loss (ASDS loss)
python svg_render.py x=diffsketcher "prompt='an elephant. minimal 2d line drawing. trending on artstation.'" x.token_ind=2 x.sds.grad_scale=1 x.sds.num_aug=4 x.clip.vis_loss=0 x.perceptual.coeff=0 x.opacity_delta=0.3 
# Vectorized Particle-based Score Distillation (VPSD loss)
python svg_render.py x=svgdreamer "prompt='a panda rowing a boat in a pond. minimal flat 2d vector icon. lineal color. trending on artstation.'" save_step=60 x.guidance.n_particle=6 x.guidance.vsd_n_particle=4 x.guidance.phi_n_particle=2 

❓ FAQ

^{▴ Back to top}

• Q: Where can I get more scripts and visualizations?

• A: Check the pytorch-svgrender.readthedocs.io.

• Q: An error says HuggingFace cannot find the model in the disk cache.

• A: Add diffuser.download=True to the command for downloading model checkpoints the first time you run the script.

• Q: It says xFormers is not built with CUDA support or xFormers cannot load C++/CUDA extensions.

• A: You need to install xFormers again using the command pip install --pre -U xformers instead of the conda one.

🗒 TODO

^{▴ Back to top}

• SVG Layout Pipeline: Given words and layout attrs, returning layout in SVG format.
• SVGDreamer supports fp16 optimization.
• integrated SVGDreamer.

🤝 Contribution

• How to Contribute to the PyTorch-SVGRender Community?

• We greatly appreciate your contributions to the PyTorch-SVGRender community and helping us make it even better than it is now! For more details, please refer to the Contribution Guidelines.

  Remember to use test/test_svgrender.py to test your code before committing it.

💘 Acknowledgement

^{▴ Back to top}

The project is built based on the following repository:

BachiLi/diffvg, huggingface/diffusers, threestudio-project/threestudio, yael-vinker/CLIPasso, ximinng/DiffSketcher, THUDM/ImageReward, advimman/lama

We gratefully thank the authors for their wonderful works.

📚 Citation

^{▴ Back to top}

If you use this code for your research, please cite the following work:

@InProceedings{xing2024svgdreamer,
    author    = {Xing, Ximing and Zhou, Haitao and Wang, Chuang and Zhang, Jing and Xu, Dong and Yu, Qian},
    title     = {SVGDreamer: Text Guided SVG Generation with Diffusion Model},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {4546-4555}
}
@inproceedings{xing2023diffsketcher,
    title={DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models},
    author={XiMing Xing and Chuang Wang and Haitao Zhou and Jing Zhang and Qian Yu and Dong Xu},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS)},
    year={2023},
    url={https://openreview.net/forum?id=CY1xatvEQj}
}

Licence

^{▴ Back to top}

This work is licensed under a Mozilla Public License Version 2.0.