Modifying unet architecture

[junsukha]

Hi! thank you for sharing the work.

I would appreciate if you could give some advice on how to modify zero123's unet architecture.

I want to add a custom layer to zero123's unet module. I think it would be convenient if I could use diffusers library.

I see that you load a unet model by using from_pretrained.
Suppose I did load a unet model. How can I add a custom layer to this loaded unet model? Do I have to write a function that does this or is there an easier way like by using diffuers lib functions?

Basically, I want to make a custom unet model based off of a loaded unet model using from_pretrained.

FYI, this author wrote this function to use one's own unet model.
https://github.com/facebookresearch/ViewDiff/blob/7df36e54728f760a119dac91a1fad01fab0a71aa/viewdiff/model/custom_unet_2d_condition.py#L809

Is that a typical way to achieve my goal?

Thanks!

[kxhit]

Hi @junsukha , you can similarly achieve that by loading your local customized unet_2d_condition module like here. Basically, you can use your local diffusers lib and modify/add new customized layers in your local.