Flux how to use the lora model with T5xxl trained
Opened this issue · 3 comments
I have a model trained with T5-xxl = True, which is a 827M model (rank=64). When I pass this model into the WebUI Forge, I see the following logs:
[LORA] Loaded /ai_model/dev/comfyui/models/loras/flux_with_te-step00000010.safetensors for KModel-UNet with 304 keys at weight 1.0 (skipped 0 keys) with on_the_fly = False
[LORA] Loaded /ai_model/dev/comfyui/models/loras/flux_with_te-step00000010.safetensors for KModel-CLIP with 240 keys at weight 1.0 (skipped 0 keys) with on_the_fly = False
Does this mean the T5 weights are not working at all? How can I properly use this LoRA?
ComfyUI seems to support LoRA with T5XXL. I'm not sure about Forge. It may be necessary to add functionality on the Forge side.
ComfyUI seems to support LoRA with T5XXL. I'm not sure about Forge. It may be necessary to add functionality on the Forge side.
Why not consider saving the LoRA or the complete matrix for T5 and CLIP separately when saving? This way, they can be directly replaced during inference, making it more convenient to use. Additionally, considering the significant difference in parameter sizes between T5 and CLIP, can it be understood that training T5 is substantially more impactful than training CLIP?
It would be helpful to include the option to save the modified CLIP_L separately as well, since merging a changed clip L from a lora is kind of intrusive. I need multiple extra steps to extract and merge, and doing this autonomously would be quite convenient.