StyleGan3 Port?

[3DTOPO]

I was not able to find a StyleGan3 version of this project so I gave it a shot but got stuck because the StyleGan 2 and 3 models are apparently quite different.

For instance, the forward function for StyleGan 3 takes two tensors 'z' and 'c'

 def forward(self, z, c, truncation_cutoff=None, update_emas=False):

While StyleGan 2 it only takes one tensor 'styles':

def forward(
    self,
    styles,
    return_latents=False,
    inject_index=None,
    truncation=1,
    truncation_latent=None,
    input_is_latent=False,
    input_is_s_code=False,
    noise=None,
    randomize_noise=True,
):

Any suggestions? This repo seems quite powerful and would be nice if could loose the old TF support.

[rinongal]

Hi! We actually already have a StyleGAN3 branch!

Keep in mind that:

1. It doesn't have support for automatic layer freezing and I didn't re-implement the StyleCLIP mapper for SG3, so its only got partial functionality.
2. Some StyleGAN3 optimizations cause the learned grid to pop out. I tried to minimize the effect but completely removing it will likely take some more serious adjustments.

[3DTOPO]

Wow, thank you! I don't know how I missed that - I did do some searching.

I have some experience hooking clip up to loss functions so I will take a look to see what I can do.

I'll look into freezing. Maybe I can use StyleGan3's freezing features? E.g. for training you can freeze X layers.

Thanks again!

[rinongal]

Happy to help.

There's already code there for manual layer freezing, so you can select which layers you want to train. What I didn't adapt is the automatic layer selection method since it relies on StyleCLIP and that needs a bit of adapting for SG3 as well (things like needing to make sure the optimization doesn't move the image).

Either way, looks like this is resolved so I'm closing it for now. Feel free to reopen if you need more help.

[3DTOPO]

It states it depends on the Rosinality PyTorch StyleGan2, so I am a little confused how to set it up with StyleGan3?

[rinongal]

Haa, sorry, it might not be updated correctly in the readme. You can check the stylegan3_nada notebook for an example. You need to set the --sg3 flag to let it know you're using a SG3 model.

[3DTOPO]

Super, thanks! I didn't see that file there. I should have looked better.

[rinongal]

It's all good :) We didn't exactly advertise the branch or highlight the new files either.

[3DTOPO]

You're most kind. One question, I see that SG3 inversions are not implemented. Is there a technical challenge or do you think it would be rather straightforward to port?

[rinongal]

I created that branch shortly after the SG3 release, and there was no working inversion option at the time.

I'm not sure what the current status is. I assume that if it's easy to do, someone should have a servicable inversion method available on github / colab. I'd start with optimization methods if you decide to implement something yourself.