lllyasviel/ControlNet-v1-1-nightly

Joint training of ControlNets with different conditions

andreemic opened this issue · 4 comments

andreemic commented

Hey! Thanks for all the amazing work on this project @lllyasviel.

We at Virtual Staging AI are experimenting with modifying ControlNet for virtual staging (adding furniture to empty rooms). Here's an example input/output pair using a 3D reconstruction model.
Screenshot 2023-04-02 at 13 13 00

With ControlNet we've achieved fairly good results by superimposing M-LSD lines on top of the original room image
Obviously you can get powerful results by combining multiple conditions as described in the ControlNet article on Hugging Face but does it also make sense to jointly train ControlNets with different conditions?

For example, train with two conditions as input:

  • RGB and
  • depth map

If so, is there code to train multiple ControlNets jointly? Or one ControlNet on multiple conditions?

Semolce commented

Diffusers have already enabled multi-control. I think you can train a multiple controlnets based on this script train_controlnet.py with a little modification, saying change the controlnet it use to multi controlnet and I think it will work fine. Here's an example of how to enable multi controlnetmulti-control

lllyasviel commented

Thanks for the issue. Multi-controlnet is a1111-only now.

hubble-bubble commented

Hey! Thanks for all the amazing work on this project @lllyasviel.

We at Virtual Staging AI are experimenting with modifying ControlNet for virtual staging (adding furniture to empty rooms). Here's an example input/output pair using a different model. Screenshot 2023-04-02 at 13 13 00

With ControlNet we've achieved fairly good results by superimposing M-LSD lines on top of the original room image Obviously you can get powerful results by combining multiple conditions as described in the ControlNet article on Hugging Face but does it also make sense to jointly train ControlNets with different conditions?

For example, train with two conditions as input:

  • raw image of a room and
  • depth map of the room
    So the model learns to keep the exact room pixels from the raw image and learns to insert furniture sensibly from the depth map.

If so, is there code to train multiple ControlNets jointly? Or one ControlNet on multiple conditions?

have you find the way how to train multiple ControlNets jointly？