/sd-webui-segment-anything

Segment Anything for Stable Diffusion Webui

Primary LanguagePython

Segment Anything for Stable Diffusion WebUI

This extension aim for helping stable diffusion webui users to use segment anything to do stable diffusion inpainting.

News

  • 2023/04/12: [Feature] Mask expansion enabled. Thanks @jordan-barrett-jm for your great contribution!
  • 2023/04/13: [Feature] Preview version of GroundingDINO support released in GroundingDINO branch. The full featured GroundingDINO support is still under development.

Plan

Thanks for suggestions from GitHub Issues, reddit and bilibili to make this extension better.

  • Support mask expansion
  • Support text->detection->segmentation from Grounded Segment Anything and merge to master branch
  • Batch process to support LoRA/LyCORIS training
  • Support API as mentioned in #15
  • Connect to ControlNet
  • Support WebUI older commits (e.g. a9fed7c364061ae6efb37f797b6b522cb3cf7aa2)

Update your WebUI version

If you are unable to add dot, observe list index out of range error on your terminal, or any other error, the most probable reason is that your WebUI is outdated (such as you are using this commitment: a9fed7c364061ae6efb37f797b6b522cb3cf7aa2).

In most cases, updating your WebUI can solve your problem. Before you submit your issue and before I release support for some old version of WebUI, I ask that you firstly check your version of your WebUI.

How to use

Step 1:

Download this extension to ${sd-webui}/extensions use whatever way you like (git clone or install from UI)

Step 2:

Download segment-anything model from here to ${sd-webui}/models/sam. Do not change model name, otherwise this extension may fail due to a bug inside segment anything.

To give you a reference, vit_h is 2.56GB, vit_l is 1.25GB, vit_b is 375MB. I myself tested vit_h on NVIDIA 3090 Ti which is good. If you encounter VRAM problem, you should switch to smaller models.

Step 3:

  • Launch webui and switch to img2img mode.
  • Upload your image and add prompts on the image (MUST). Left click for positive prompt (black dot), right click for negative prompt (red dot), left click any dot again to cancel the prompt. If you forgot to add prompts, there will be exceptions on your terminal.
  • Click Preview Segmentation button
  • Choose your favorite segmentation and check Copy to Inpaint Upload
  • Optionally check Expand Mask and specify the amount, then click Update Mask
  • Switch to Inpaint upload. There is no need to upload another image or mask, just leave them blank. Write your prompt, configurate and click Generate.

Demo

demo.mp4

Contribute

Disclaimer: I have not thoroughly tested this extension, so there might be bugs. Bear with me while I'm fixing them :)

If you encounter a bug, please submit a issue. Please at least provide your WebUI version, your extension version, your browser version, errors on your browser console log if there is any, error on your terminal log if there is any, to make sure that I can find a solution faster.

I welcome any contribution. Please submit a pull request if you want to contribute