Cannot generate background

[xingzhehe]

I have not read the arxiv so I am not sure if it intended. I tried to generate some small objects. If the image is not covered by the regions fully, there will be noise at those uncovered regions.

images = pipe(
        canvas_height=640,
        canvas_width=1408,
        regions=[
            Text2ImageRegion(0, 320, 0, 320, guidance_scale=4,
                             prompt="a dog"),
            Text2ImageRegion(0, 320, 384, 1024, guidance_scale=4,
                             prompt="a man"),
            Text2ImageRegion(0, 320, 768, 1408, guidance_scale=4,
                             prompt="a girl")
        ],
        num_inference_steps=50,
        seed=0,
    )["sample"]

[albarji]

Yes, unfortunately this is a problem of using this technique: if you do not cover the whole canvas with regions you will get a very noisy image.

This is because the diffusion is done in a latent space, and to recover the image in pixel space you need to apply a decoder network. Unfortunately, because of the way the decoder works, any value in latent space can have some influence in any pixel in the output. Therefore, if the diffusion process is not applied for some values in latent space, they will corrupt the whole pixel space with noise.

I have not tested this idea, but maybe to address this issue you could try adding some Text2ImageRegion that covers those parts of the canvas, specifying a simple prompt like "white background" or something similar.

[xingzhehe]

Thanks for clarification. Yeah a simple "background" prompt at blank area looks nice.