williamyang1991/ISCAS_Tutorial

[ISCAS 2024 Tutorial] New Era of Artificial Intelligence: Unleashing the Power of Large Models in Visual Applications

New Era of Artificial Intelligence:
Unleashing the Power of Large Models in Visual Applications

ISCAS 2024 Tutorial (selected for IEEE CASS Continuing Education)
Jiaying Liu, Wen-Huang Cheng, and Shuai Yang
Home Page

This repository prodives the slides of our ISCAS 2024 tutorial on New Era of Artificial Intelligence: Unleashing the Power of Large Models in Visual Applications

Contents

Title	Speaker	Tutorial Slides 🔥
Specializing Large Models for Domain-Specific Vision Tasks	Wen-Huang Cheng	PDF
AIGC for Image and Video Generation	Shuai Yang	PDF
Employing Diffusion Models for Low-level Vision	Jiaying Liu	PDF

Specializing Large Models for Domain-Specific Vision Tasks

Strategies for Domain-Specific Vision Tasks

• Fine-tuning
• Visual-Prompt Tuning
• Visual Instruction Tuning
• Open Problems

AIGC for Image and Video Generation

• Preliminary
• Image Diffusion Model
• Image Supporting Model
  • Pre-trained Prior
  • Image Editing
  • Customization
  • Control
• Video Diffusion Model
  • Data-driven Video Model
  • One-shot Video Model
  • Zero-shot Video Model

Employing Diffusion Models for Low-level Vision

• Diffusion for Image Restoration
  • Training-based
  • Sampling-based
• Diffusion for Low-light Enhancement
  • Human Vision
  • Machine Vision
• Unified Image Restoration and Enhancement

Awesome Techniques Introduced in the Tutorial

Fine-tuning

• Side-Tuning (ECCV20)
• TinyTL (NeurIPS20)
• MoCo v3 (ICCV21)
• VisE (ICCV21)

Visual-Prompt Tuning

• VPT (ECCV22)
• CoOp (IJCV22)
• CoCoOp (CVPR22)
• TaskRes (CVPR23)

Visual Instruction Tuning

• EmoVIT (CVPR24)
• EmoSet (ICCV23)
• InstructBLIP (NeurIPS23)

Open Problems

Hallucination

• A Survey on Hallucination

Long-context Modeling

• BLIVA (AAAI24)
• Leave No Context Behind

Robust Automatic Evaluation

• CLIPScore (EMNLP21)

Preliminary

• DDPM (NeurIPS20)
• DDIM (ICLR21)
• Classifier guidance (NeurIPS21)
• Classifier-free guidance (NeurIPSW21)

Image Diffusion Model

DALLE Series

• DALLE2
• DALLE3

Stable Diffusion Series

• Latent Diffusion Model (CVPR22)
• SD 1.x
• SD 2.x
• SD XL (ICLR24)
• SD XL Turbo
• SD Cascade (ICLR24)
• SD 3

Others

• Midjourney

Image Supporting Model

Pre-trained Prior

• Repaint (CVPR22)
• SDEdit (CVPR22)

Image Editing

• Prompt-to-Promt (ICLR23)
• Plug-and-Play (CVPR23)
• Null-text Inversion (CVPR23)

Model Customization

• Textual Inversion (ICLR23)
• Dream Booth (CVPR23)
• LoRA (ICLR22)
• Imagic (CVPR23)

Adding Control

• ControlNet (ICCV23)
• T2I Adapter (AAAI24)

Video Diffusion Model

Data-driven Video Model

• VDM (NeurIPS22)
• Make-A-Video (ICLR23)
• Imagen Video
• Video LDM (CVPR23)
• HD-VG-130M (dataset)
• Panda-70M (CVPR24, dataset)
• Sora

One-shot Video Model

• Tune-A-Video (ICCV23)
• Edit-A-Video (ACML23)
• Video-P2P (CVPR24)

Zero-shot Video Model

Inversion-free model

• Text2Video-Zero (ICCV23)
• ControlVideo (ICLR24)
• Rerender-A-Video (SIGGRAPH Asia23)
• FRESCO (CVPR24)
• Fairy (CVPR24)

Inversion-based model

• FateZero (ICCV23)
• Pixel2Video (ICCV23)
• TokenFlow (ICLR24)
• FLATTEN (ICLR24)
• VidToMe (CVPR24)

Diffusion for Image Restoration

Training-based

• SR3 (TPAMI22)
• StableSR
• DiffIR (ICCV23)

Sampling-based

• DDRM (NeurIPS22)
• DiffPIR (CVPRW23)
• GDP (CVPR23)
• DPS (ICLR23)
• BoostDiffSR (ICLR24)

Diffusion for Low-light Enhancement

Human Vision

Acceleration

• PyDiff (IJCAI23)
• DiffLL (TOG23)

Interpretability

• Diff-Retinex (ICCV23)
• ExposureDiffusion (ICCV23)
• QuadPrior (CVPR24)

Machine Vision

• DiD (WACV24)

Unified Image Restoration and Enhancement

• Palette (SIGGRAPH22)
• UCDIR (NeurIPS23)
• GDP (CVPR23)

Acknowledgments

Thanks to Wenjing Wang, Yiyang Ma, Zejia Fan and Haowei Kuang for their supports during the preparation of the slides.