[Feature Request]Support Dora

[sdbds]

paper:https://arxiv.org/pdf/2402.09353.pdf
code:https://github.com/catid/dora/blob/main/dora.py

[Visual-Synthesizer]

Another implementation https://github.com/rasbt/dora-from-scratch/blob/main/lora-dora-mlp.ipynb

[gesen2egee]

it had been supported by lycoris dev?

[Visual-Synthesizer]

Where supported?

[SlZeroth]

+1

[SlZeroth]

@sdbds did you solve it? I successfully trained dora. but I cant inference with it, because webui and diffusers not support load dora

[gesen2egee]

train:
kohya requestment.txt lycoris_lora==2.2.0
--network_args "dora_wd=True"

inference:
webui use this PR
AUTOMATIC1111/stable-diffusion-webui#15239

You can replace it
https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/9f2ae1cb85015ee3cc79f1de92641032df154d5b/extensions-builtin/Lora/network.py

[SlZeroth]

--network_module=lycoris.kohya --network_args "algo=lora" "dora_wd=True" is it right?
because I got error in webui "Networks with errors: lora (956)" @gesen2egee

[gesen2egee]

try "algo=locon"? but it may alias

[gesen2egee]

webui? did you replace the network.py

[SlZeroth]

yes I replaced to https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/9f2ae1cb85015ee3cc79f1de92641032df154d5b/extensions-builtin/Lora/network.py but webui got error

I dont know about reason why. I trying to train again with -network_args "algo=locon" "dora_wd=True" now

[gesen2egee]

I also dont know about reason why.
If you still have problem
Can join Lycoris DC
https://discord.gg/VtTFKrj9gJ

[NineMeowICT]

I dont know about reason why. I trying to train again with -network_args "algo=locon" "dora_wd=True" now

@SlZeroth Did you succeed?