AUTOMATIC1111/stable-diffusion-webui

[Feature Request]: step by step diffusion

Ehplodor opened this issue · 7 comments

Ehplodor commented

Is there an existing issue for this?

  • I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Sometimes, i get the feeling that diffusion is going in the right direction, preview images are good for a few steps, then some steps later the preview is completely unrecognizable. That is quite disturbing. I believe the sheduled step is too high, and that it crushes the current latents. Therefore, I would like to run diffusion step by step, recording latents in the meantime, in order to be able to come back to an earlier version and altering the scheduling (lowering its value dynamically) in order to better control the diffusion process.

Proposed workflow

  1. Go to img2img
  2. activate "1-step diffusion" option
  3. click "run 1-step" ... n times
  4. if needed, adjust diffusion parameters (CFG, denoising strength, ?scheduler?, prompt, ...)
  5. continue until the end or until user decides to stop.

Additional information

Relevant (same subject) discussion "image Generate debugger" proposed by @aifartist : #3396

Issue showing sometimes (last image) the step is not optimal, even undesirable : #3483

Here, sometimes the preview images do not change much and it could be desireable to effectively stop generation earlier : #2472

Further discussion about steps in general : #1113 (comment)

ClashSAN commented

I think this sort of thing should be built on the "process_one" commit by @macrosoft
a9e9799
I do not know any scripts/extensions using it yet.

Ehplodor commented

I think this sort of thing should be built on the "process_one" commit by @macrosoft a9e9799 I do not know any scripts/extensions using it yet.

TY @ClashSAN I knew there had been some work in this direction but couldn't find it when searching for relevant info.

toriato commented 
import os

from modules import scripts
from modules.sd_samplers import KDiffusionSampler, sample_to_image

orig_callback_state = KDiffusionSampler.callback_state


def callback_state(self: KDiffusionSampler, d: dict) -> None:
    '''
    callback from k-diffusion sampling methods
    https://github.com/crowsonkb/k-diffusion/blob/60e5042ca0da89c14d1dd59d73883280f8fce991/k_diffusion/sampling.py#L67
    '''
    current_step = d['i']
    latent = d["denoised"]

    image = sample_to_image(latent)
    image.save(os.path.join(scripts.basedir(), f"step_{current_step}.png"))

    return orig_callback_state(self, d)


setattr(KDiffusionSampler, "callback_state", callback_state)


class Script (scripts.Script):
    def title(self):
        return "Step by step"

    def show(self, _):
        return scripts.AlwaysVisible

This code might be helpful to make extension.

Ehplodor commented

TY. I looked at your example and other scripts as well for more info however I just don't understand most of it. Frankly speaking, I would really like to help but I can't with my present knowledge. Else I would already have made a PR for this feature as well a a few others. All I can do for the moment is, just like most of us that are more "user" than "devs", give some brain-time to propose ideas that could eventually be implemented, or maybe not ;-) Hope that helps, too.

aifartist commented
aifartist commented