dfaker/SD-latent-mirroring

How feasible is to add Dynamic Symmetry to SD-Latent-Mirroring?

Kantuva opened this issue · 2 comments

Kantuva commented

Hey guys, not an engineer, but an artist

Found this repo from a discussion yesterday about the lack of Dynamic Symmetry being used by StableDiffusion

I was quite abrasive on that discussion so please forgive me for that, but, it is very much something that I would love to have, as I use it daily on my job and work, having SD "somehow" use the armatures of Dynamic Symmetry would be just incredible, as Midjourney makes strong use of it already, and it is one of the strong qualities which make Midjourney be stronger on the technical aesthetics side than SD currently is

Here's an example from a classical painting teacher showing what Dynamic Symmetry is and how it is used in paintings

Having a way to control the "fit" of shapes, and diagonals alongside the lines of Dynamic Symmetry would be indeed amazing, but I just dont know how possible it even is! So would be interested on this discussion and what you guys think about it

If you need SVG files to use let me know.

Here are some links for looking around if interested.

(Using Archive links for archival purposes because the website might be shutting down in some days from now 😔)

Anyhow, would love to hear what you guys might have to say about this, as I dont really have got the backdrop to know the feasibility of it

dfaker commented

Two possible approaches spring to mind, I'm not sure how we'll they'd in work in reality, in effect you'd be choosing to either:

A> Apply regional symmetry across sections of the image, guided perhaps by your bisection image:

image

Mirroring details from blue-red and green-magenta perhaps with some affine transformation and falloff at the edges, but there's a clear issue with overlapping mirrored regions the areas where two differing mirrored regions meet would most likely not by stylistically consistent.

B> Supressing the latents guided by a mask:

image

Would constrain any refinement of detail to the white section, but how the network would react to that border and the method of supressing the latents is up in the air.

Kantuva commented

Hmmmm, again, I am not sure how this might do implementation wise

But I had the idea of as well, color coding the important diagonals on the symmetry all the same color, so "code wise" you might tell it, "if X shape fits close to a colored diagonal, other shapes ought also go alongside said colored diagonals or similarly diagonals by how close they are numerically to the initial color"

DS PS Example

So, lets say, an arm ends up being near the red direction of the diagonal, lets say, some branches of trees in the background would also want to go in the same direction as the arm because they share the same diagonal color

Then you could do a Gaussian blur pass and you might arrive to something like this

DS PS Example Gausian blur smaller
The idea of blurring at least in my mind would be to create a sorts of voroni styled field/heatmap, where shapes that are being generated "want" to go alongside" the diagonal like a magnetic field of sorts, and trying to imitate what is happening on this midjourneyV3 made image with the "arms" of these creatures

blabla

For example, look at the "rail" in Cyan at the bottom left, that "railway" thing goes very close to the actual line of the armature, as if it when it was being diffused, it had a sorts of "magnetism" where it wanted to fit onto place where the armature was nearly like a heatmap

Again, apologies for the layman terminology, but that's at least how I could think of it maybe happening, the armature diagonals working as a literal heatmap where shapes like to coalesce around, which is pretty much how these armatures are used in real life

So far as actual implementation, please understand that painters are used to convoluted tools, so it is fine if things are odd to use so long as it is feasible! Then it is all fine

Anyhow, thank you for sparing the thought on my ramblings

//EDIT
Another composition example in real life by Nicboone on twitter, look how arms, legs, even the wheel of the car, or that small dark gap in between the panels of the car are all fitted to the actual dynamic symmetry armature, all of these details are to a degree or another purposeful, the arms at high also align with the direction of the diagonals in the same manner as the legs do, or the door giving a rough direction as well

https://twitter.com/NicBoone/status/1604884273269379078
Composition example