Issue with hiresfix
Closed this issue · 6 comments
img2img and hiresfix use the encoder block of the VAE. Possible apply the inverse operation to the first layer of the encoder? Will need to do testing
Was just about to open this issue, the VAE provided here in this repo and the on-the-fly modification via this extension have the same problem.
Fixed the problem, will update this repo and make a PR on the extension soon
Left is with the fix and right is without
No hiresfix
R-ESRGAN 4x+ Anime6B hires upscaler
Notice how the old (right) loses contrast again from passing through the VAE. However, by implying the inverse operation to the VAEs encoder, we can preserve the contrast.
However, this method isn't perfect. We are actually modifying the latents themselves this way. This can cause the images to not look as good. However, the effect shouldn't be noticable because we are going to add a ton of noise to the latents anyways. Note that this will not affect anything besides hiresfix and img2img.
Just passing the image into the VAE and back out destroys a bit of quality. However, I don't see any use just passing images through a VAE, generally they are being used with denoising as well, so the quality difference shouldn't make a noticeable impact.
Isn't brightness now applying the same operation as contrast? bias is only mentioned in the comment and not actually used in the code now. Actually it looks like the brightness code is copied from contrast verbatim, including variables.
yeah.......... just fixed that 😓