A diffusers based implementation of HyperDreamBooth
the code is based on the HyperDreamBooth implementation in the LyCORIS project.
This section is a brief introduction of HyperDreamBooth, I will split hyperdreambooth into 4 sections. And this projection will have 4 corresponding scripts
Before we start to train the hypernetwork, we should train the lora(lilora) on each identites(instance) first. In this implementation, we took a batch of instances and then do a inner training loop to get the pre optimized weights.
Just train it. send the image into hypernetwork, get the weights, apply to unet. Calc the loss based on diffusion loss and weight loss.
Use the image of the identity you want to train on, generate the weight from hypernetwork.
resume from the generated weight, do a few step training (for about 20~50step)