A collection of sketch based applications.
Feel free to create a PR or an issue.
Outline
- 0. Survey
- 1. Sketch Based Image Synthesis
- 2. Sketch Based Image Editing
- 3. Sketch Based Image Retrieval (SBIR)
- 4. Sketch Based 3D Shape Retrieval
- 5. Sketch Based 3D Shape Modeling
- 6. Sketch Based Garment Design
- 7. Sketch Completion
- 8. Sketch Restoration, Retargeting and Inpainting
- 9. Sketch and Shading
- 10. Sketch Enhancement
- 11. Sketch-Guided Object Localization
- 12. Sketch-Guided Video Synthesis
- 13. Sketch Recognition
- 14. Sketch Segmentation and Perceptual Grouping
- 15. Sketch Representation Learning
- 16. Sketch and Visual Correspondence
- 17. Sketch Animation/Inbetweening
- 18. Sketch and AR/VR
- 19. Sketch Based Incremental Learning
- 20. Sketch Quality Measurement
- 21. Cloud Augmentation with Sketches
- 22. Sketch and Re-identification
- 23. Sketch-based Salient Object Detection
Paper | Source | Code/Project Link |
---|---|---|
Deep Learning for Free-Hand Sketch: A Survey | TPAMI 2022 | [code] |
- Natural Image or Object
- Human Face / Portrait
- 3D image
Paper | Source | Code/Project Link |
---|---|---|
3D-aware Conditional Image Synthesis | arxiv 23.02 | [project] [code] |
Paper | Source | Code/Project Link |
---|---|---|
TextureGAN: Controlling Deep Image Synthesis with Texture Patches | CVPR 2018 | [code] |
Multimodal Unsupervised Image-to-Image Translation | ECCV 2018 | [code] |
SketchPatch: Sketch Stylization via Seamless Patch-level Synthesis | SIGGRAPH Asia 2020 | |
Self-Supervised Sketch-to-Image Synthesis | AAAI 2021 | [code] |
CoGS: Controllable Generation and Search from Sketch and Style | ECCV 2022 | |
Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model | WACV 2023 | [project] [code] |
Paper | Source | Code/Project Link |
---|---|---|
Sketch-Guided Text-to-Image Diffusion Models | SIGGRAPH 2023 | [project] |
Adding Conditional Control to Text-to-Image Diffusion Models | ICCV 2023 | [code] |
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models | arxiv 23.02 | [code] |
UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild | arxiv 23.05 | [code] [project] |
Stable Doodle (ClipDrop) | [demo] |
- Arbitrary Image
Paper | Source | Code/Project Link |
---|---|---|
Free-Form Image Inpainting with Gated Convolution (DeepFill v2) | ICCV 2019 | [project] [code] |
Fashion Editing with Multi-scale Attention Normalization | 1906.00884 | |
DeFLOCNet: Deep Image Editing via Flexible Low-level Controls | CVPR 2021 | [code] |
DeepSIM: Image Shape Manipulation from a Single Augmented Training Sample | ICCV 2021 | [code] [project] |
SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches | CVPR 2022 | [code] [project] |
Draw2Edit: Mask-Free Sketch-Guided Image Manipulation | ACM MM 2023 | [code] |
- Human Face / Portrait / Hair
- Anime Editing
Paper | Source | Code/Project Link |
---|---|---|
AniFaceDrawing: Anime Portrait Exploration during Your Sketching | SIGGRAPH 2023 | [project] |
- Video Editing
Paper | Source | Code/Project Link |
---|---|---|
DeepFaceVideoEditing: Sketch-based Deep Editing of Face Videos | SIGGRAPH 2022 | [project] |
- Object-level
- Scene-level
- Video Retrieval
Paper | Source | Code/Project Link |
---|---|---|
Fine-Grained Instance-Level Sketch-Based Video Retrieval | TCSVT 2020 | |
Fine-Grained Video Retrieval with Scene Sketches | TIP 2023 | [project] |
- Free-hand sketch
- NeRF
Paper | Source | Code/Project Link |
---|---|---|
SKED: Sketch-guided Text-based 3D Editing | ICCV 2023 | [project] |
- CAD sketch
Paper | Source | Code/Project Link |
---|---|---|
Lifting Freehand Concept Sketches into 3D | SIGGRAPH Asia 2020 | [project] [code] |
Sketch2CAD: Sequential CAD Modeling by Sketching in Context | SIGGRAPH Asia 2020 | [project] [code] |
Free2CAD: Parsing Freehand Drawings into CAD Commands | SIGGRAPH 2022 | [project] [code] |
Symmetry-driven 3D Reconstruction From Concept Sketches | SIGGRAPH 2022 | [project] |
Piecewise-smooth Surface Fitting Onto Unstructured 3D Sketches | SIGGRAPH 2022 | [project] |
Reconstruction of Machine-Made Shapes from Bitmap Sketches | SIGGRAPH Asia 2023 | [project] |
- Sketch completion
- Sketch gap / connectivity detection
Paper | Source | Code/Project Link |
---|---|---|
Joint Gap Detection and Inpainting of Line Drawings | CVPR 2017 | [project] [code] |
Detecting Viewer-Perceived Intended Vector Sketch Connectivity | SIGGRAPH 2022 | [project] |
- Manga
Paper | Source | Code/Project Link |
---|---|---|
Exploiting Aliasing for Manga Restoration | CVPR 2021 | [webpage] [code] |
Seamless Manga Inpainting with Semantics Awareness | SIGGRAPH 2021 | [webpage] [code] |
Screentone-Preserved Manga Retargeting | arxiv 22.03 | |
Manga Rescreening with Interpretable Screentone Representation | arxiv 23.06 |
- Hand-Drawn Drawings
Paper | Source | Code/Project Link |
---|---|---|
Restoration of Hand-Drawn Architectural Drawings using Latent Space Mapping with Degradation Generator | CVPR 2023 |
Paper | Source | Code/Project Link |
---|---|---|
Learning to Shadow Hand-drawn Sketches | CVPR 2020 | [project] [code] |
SmartShadow: Artistic Shadow Drawing Tool for Line Drawings | ICCV 2021 | [project] |
Paper | Source | Code/Project Link |
---|---|---|
SketchMan: Learning to Create Professional Sketches | ACM MM 2020 | [code] |
Paper | Source | Code/Project Link |
---|---|---|
Sketch-Guided Object Localization in Natural Images | ECCV 2020 | [project] [code] |
Localizing Infinity-shaped fishes: Sketch-guided object localization in the wild | arxiv 21.09 | [code] |
Query-guided Attention in Vision Transformers for Localizing Objects Using a Single Sketch | arxiv 23.03 | |
What Can Human Sketches Do for Object Detection? | CVPR 2023 | [project] |
Sketch-based Video Object Segmentation: Benchmark and Analysis | BMVC 2023 | [code] |
Paper | Source | Code/Project Link |
---|---|---|
Deep Sketch-guided Cartoon Video Inbetweening | TVCG 2021 | [code] |
- Semantic / Instance Segmentation
- Panoptic Segmentation
Type | Paper | Source | Code/Project Link |
Vector | FloorPlanCAD: A Large-Scale CAD Drawing Dataset for Panoptic Symbol Spotting | ICCV 2021 | [project] |
GAT-CADNet: Graph Attention Network for Panoptic Symbol Spotting in CAD Drawings | CVPR 2022 | ||
CADTransformer: Panoptic Symbol Spotting Transformer for CAD Drawings | CVPR 2022 | [code] | |
VectorFloorSeg: Two-Stream Graph Attention Network for Vectorized Roughcast Floorplan Segmentation | CVPR 2023 | [code] |
- Perceptual Grouping
Type | Paper | Source | Code/Project Link |
Stroke-level | Sketching by perceptual grouping | ICIP 2013 | |
Making Better Use of Edges via Perceptual Grouping | CVPR 2015 | ||
Universal Sketch Perceptual Grouping / Toward Deep Universal Sketch Perceptual Grouper | ECCV 2018 / TIP 2019 | [code] |
- Stroke order importance/saliency, sketch abstraction
Paper | Source | Code/Project Link | Data Manner | Related Applications |
---|---|---|---|---|
Learning Deep Sketch Abstraction | CVPR 2018 | Vector/stroke-level | FG-SBIR | |
Goal-Driven Sequential Data Abstraction | ICCV 2019 | Vector/stroke-level | Sketch recognition | |
Pixelor: a competitive sketching AI agent. So you think you can sketch? | SIGGRAPH Asia 2020 | [Project] [Code] | Vector/stroke-level | Sketch synthesis and recognition |
SketchAA: Abstract Representation for Abstract Sketches | ICCV 2021 | Vector/stroke-level | Sketch recognition, FB-SBIR, sketch healing | |
CLIPasso: Semantically-Aware Object Sketching | SIGGRAPH 2022 | [project] [code] | Vector/stroke-level | |
Abstracting Sketches through Simple Primitives | ECCV 2022 | [code] | Vector/stroke-level | Sketch recognition, FG-SBIR |
Learning Geometry-aware Representations by Sketching | CVPR 2023 | [code] | Vector/stroke-level | object attribute classification, domain transfer, stroke-based generation, FG-SBIR |
SketchXAI: A First Look at Explainability for Human Sketches | CVPR 2023 | [project] | Vector/stroke-level | sketch recognition |
Prediction with Visual Evidence: Sketch Classification Explanation via Stroke-Level Attributions | TIP 2023 | Vector/stroke-level |
- Conventional Representation Learning
Paper | Source | Code/Project Link | Data Manner | Related Applications |
---|---|---|---|---|
A Neural Representation of Sketch Drawings (Sketch-RNN) | ICLR 2018 | [code] | Vector/stroke-level | Reconstruction and interpolation |
SketchHealer: A Graph-to-Sequence Network for Recreating Partial Human Sketches | BMVC 2020 | [code] | Vector/stroke-level | Sketch recognition, retrieval, completion and analogy |
Sketchformer: Transformer-based Representation for Sketched Structure | CVPR 2020 | [code] | Vector/stroke-level | Sketch classification, SBIR, reconstruction and interpolation |
CoSE: Compositional Stroke Embeddings | NeurIPS 2020 | [Code] | Vector/stroke-level | Auto-completing diagrams |
SketchLattice: Latticed Representation for Sketch Manipulation | ICCV 2021 | [code] | Lattice graph | Sketch healing and image-to-sketch synthesis |
SketchODE: Learning neural sketch representation in continuous time | ICLR 2022 | [Project] | Vector/Stroke-level | Continuous-time representation, reconstruction & interpolation |
Linking Sketch Patches by Learning Synonymous Proximity for Graphic Sketch Representation | AAAI 2023 | [code] | Vector/Stroke-level | Sketch synthesis and sketch healing |
SketchKnitter: Vectorized Sketch Generation with Diffusion Models | ICLR 2023 | [code] | Vector/Stroke-level | vectorized sketch generation |
ChiroDiff: Modelling chirographic data with Diffusion Models | ICLR 2023 | [Project] | Vector/Stroke-level | vectorization, de-noising/healing, abstraction |
Paper | Source | Code/Project Link | Data Manner | Related Applications |
---|---|---|---|---|
A Learned Representation for Scalable Vector Graphics | ICCV 2019 | [code] | SVG | Font design |
DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation | NeurIPS 2020 | [Code] [Project] | SVG | Vector Graphics Animation, reconstruction and interpolation |
SVGformer: Representation Learning for Continuous Vector Graphics using Transformers | CVPR 2023 | SVG | Reconstruction, classification, interpolation, retrieval |
- Self-supervised or few/zero-shot learning
Paper | Source | Code/Project Link | Data Manner | Related Applications |
---|---|---|---|---|
Sketch Fewer to Recognize More by Learning a Co-Regularized Sparse Representation | TCSVT 2019 | few-shot classification | ||
On Learning Semantic Representations for Large-Scale Abstract Sketches | TCSVT 2020 | [Code] | Bitmap and Vector | Retrieval and recognition |
Deep Self-Supervised Representation Learning for Free-Hand Sketch | TCSVT 2020 | [Code] | Vector/stroke-level | Retrieval and recognition |
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt | CVPR 2020 | [Code] | Vector/stroke-level | Sketch recognition, retrieval, and gestalt |
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting | CVPR 2021 | [Code] | Both Vector and Raster-level | Recognition |
- Scene-level
Paper | Source | Code/Project Link |
---|---|---|
SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text | CVPR 2023 |
- Datasets
Name | Paper | Source | Code/Project Link |
---|---|---|---|
CreativeFlow+ | Creative Flow+ Dataset | CVPR 2019 | [code] |
ATD-12K | Deep animation video interpolation in the wild | CVPR 2021 | [code] |
AnimeRun | AnimeRun: 2D Animation Correspondence from Open Source 3D Movies | NeurIPS 2022 | [code] |
- Methods
Paper | Source | Code/Project Link |
---|---|---|
Globally optimal toon tracking | SIGGRAPH 2016 | [project] |
SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence | TCSVT 2020 | |
SketchZooms: Deep Multi-view Descriptors for Matching Line Drawings | CGF 2021 | [project] |
The Animation Transformer: Visual Correspondence via Segment Matching | ICCV 2021 | |
Differential Operators on Sketches via Alpha Contours | SIGGRAPH 2023 | [code] |
- Inbetweening
Paper | Source | Representation | Code/Project Link |
---|---|---|---|
BetweenIT: An Interactive Tool for Tight Inbetweening | CGF 2010 | stroke | |
Context-Aware Computer Aided Inbetweening | TVCG 2017 | stroke | |
FTP-SC: Fuzzy Topology Preserving Stroke Correspondence | SCA 2018 | stroke | [webpage] [video] |
Cacani: 2d animation and inbetween software | / | stroke | [software] |
Stroke-Based Drawing and Inbetweening with Boundary Strokes | CGF 2021 | stroke | |
Deep Geometrized Cartoon Line Inbetweening | ICCV 2023 | vertex/point | [code] |
Optical Flow Based Line Drawing Frame Interpolation Using Distance Transform to Support Inbetweenings | ICIP 2019 | raster | |
Bridging the Gap: Fine-to-Coarse Sketch Interpolation Network for High-Quality Animation Sketch Inbetweening | arxiv 23.08 | raster |
- Animation
Paper | Source | Representation | Code/Project Link |
---|---|---|---|
Autocomplete Hand-drawn Animations | SIGGRAPH Asia 2015 | stroke | [webpage] [video] |
Live Sketch: Video-driven Dynamic Deformation of Static Drawings | CHI 2018 | vector | [video] |
Animating Portrait Line Drawings from a Single Face Photo and a Speech Signal | SIGGRAPH 2022 | image | [code] |
A Method for Animating Children’s Drawings of the Human Figure | TOG 2023 | image | [code] [project] [demo] |
Non-linear Rough 2D Animation using Transient Embeddings | CGF 2023 | stroke | |
Sketch Video Synthesis | arxiv 23.11 | stroke | [code] [project] |
Breathing Life Into Sketches Using Text-to-Video Priors | arxiv 23.11 | stroke | [code] [project] |
Paper | Source | Code/Project Link |
---|---|---|
SweepCanvas: Sketch-based 3D Prototyping on an RGB-D Image | UIST 2017 | [video] |
Model-Guided 3D Sketching | TVCG 2018 | [video] |
Mobi3DSketch: 3D Sketching in Mobile AR | CHI 2019 | [video] |
Interactive Liquid Splash Modeling by User Sketches | SIGGRAPH Asia 2020 | [video] |
Towards 3D VR-Sketch to 3D Shape Retrieval | 3DV 2020 | [code] [project] |
3D Curve Creation on and around Physical Objects with Mobile AR | TVCG 2021 | [video] |
HandPainter - 3D Sketching in VR with Hand-based Physical Proxy | CHI 2021 | [video] |
Fine-Grained VR Sketching: Dataset and Insights | 3DV 2021 | [code] |
Structure-Aware 3D VR Sketch to 3D Shape Retrieval | 3DV 2022 | [code] |
GestureSurface: VR Sketching through Assembling Scaffold Surface with Non-Dominant Hand | TVCG 2023 | |
3D VR Sketch Guided 3D Shape Prototyping and Exploration | ICCV 2023 | [code] |
Paper | Source | Code/Project Link |
---|---|---|
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches | CVPR 2022 | [code] |
Paper | Source | Code/Project Link |
---|---|---|
Finding Badly Drawn Bunnies | CVPR 2022 |
Paper | Source | Code/Project Link |
---|---|---|
Cloud2Sketch: Augmenting Clouds with Imaginary Sketches | ACM MM 2022 | [project] |
Paper | Source | Code/Project Link |
---|---|---|
Cross-Compatible Embedding and Semantic Consistent Feature Construction for Sketch Re-identification | ACM MM 2022 | [code] |
SketchTrans: Disentangled Prototype Learning with Transformer for Sketch-Photo Recognition | TPAMI 2023 | [code] |
Paper | Source | Code/Project Link |
---|---|---|
Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings | CVPR 2023 |