[Test] Stylize Video

[s9roll7]

1. Tile -> upscale
   motion_module : mm_sd_v14.ckpt
   steps : 20
   guidance_scale : 10
   [0]
   ip_adapter_plus ("is_plus_face": false, "is_plus": true) / scale : 0.5
   controlnet_tile / controlnet_conditioning_scale : 0.75
   size : 512x512
   context : 16
   [1]
   ip_adapter_plus / scale : 0.5
   controlnet_tile / controlnet_conditioning_scale : 1.0
   size : 1024x1024
   context : 8

style_tile_sample.mp4

2. Lineart -> upscale
   motion_module : mm_sd_v15_v2.ckpt
   steps : 20
   guidance_scale : 10
   [0]
   ip_adapter_plus / scale : 0.5
   controlnet_lineart / controlnet_conditioning_scale : 1.0
   controlnet_ip2p / controlnet_conditioning_scale : 0.5
   size : 512x512
   context : 16
   [1]
   ip_adapter_plus / scale : 0.5
   controlnet_tile / controlnet_conditioning_scale : 1.0
   controlnet_ip2p / controlnet_conditioning_scale : 0.5
   size : 768x768
   context : 8

lineart_style_sample.mp4

3. Openpose -> upscale
   motion_module : mm_sd_v15_v2.ckpt
   steps : 20
   guidance_scale : 10
   [0]
   ip_adapter_plus / scale : 0.5
   controlnet_openpose / controlnet_conditioning_scale : 1.0
   size : 512x512
   context : 16
   [1]
   ip_adapter_plus / scale : 0.5
   controlnet_tile / controlnet_conditioning_scale : 1.0
   size : 1024x1024
   context : 8

openpose_style_sample.mp4

4. Softedge -> upscale
   motion_module : mm-Stabilized_high.pth
   steps : 20
   guidance_scale : 10
   [0]
   ip_adapter_plus / scale : 0.5
   controlnet_softedge / controlnet_conditioning_scale : 1.0
   controlnet_ip2p / controlnet_conditioning_scale : 0.5
   size : 512x760
   context : 16
   [1]
   ip_adapter_plus / scale : 0.5
   controlnet_tile / controlnet_conditioning_scale : 1.0
   size : 768x1136
   context : 8

softedge_style_sample.mp4

[sdqq1234]

Why my result is bad when I use openpose,
I updated to the latest on the morning of September 13th
I used command animatediff generate -c config/prompts/Meina.json -W 424 -H 768 -L 16 -C 16
my openpose images

my result

MyPrompts
MyPrompts.zip

[s9roll7]

MyPrompts

You are trying to apply controlnet to all frames. In that case, you have to cut the effect of controlnet on the neighboring frames

#16
#6 (comment)

    "controlnet_openpose":{
      "enable": true,
      "use_preprocessor":true,
      "guess_mode":false,
      "controlnet_conditioning_scale": 1.0,
      "control_guidance_start": 0.0,
      "control_guidance_end": 1.0,
      "control_scale_list":[0.5,0.4,0.3,0.2,0.1]      # <---- ng
    },

    "controlnet_openpose":{
      "enable": true,
      "use_preprocessor":true,
      "guess_mode":false,
      "controlnet_conditioning_scale": 1.0,
      "control_guidance_start": 0.0,
      "control_guidance_end": 1.0,
      "control_scale_list":[]      # <---- ok
    },

[sdqq1234]

thank you very much, it's much better!!!

[yuturiy]

@s9roll7 can you please provide the full prompt.json for this video

style_sample.mp4

I try to reproduce but fail every time

[s9roll7]

Unfortunately, it looks like I overwrote the config file and it didn't remain.
What exactly happens when you say "fail"? The video file is not generated?

[yuturiy]

Unfortunately, it looks like I overwrote the config file and it didn't remain. What exactly happens when you say "fail"? The video file is not generated?

Sorry. By fail I mean I get very inconsistent video. So it would be nice to have a complete prompt example where everything is consistent.

[s9roll7]

Are you using lora? it is quite difficult to keep the character and style consistent without lora.

[yuturiy]

Are you using lora? it is quite difficult to keep the character and style consistent without lora.

no, there was nothing about loras near stylize examples(
will try, thanks

[tokimwc]

Hello @s9roll7

I've been exploring your repository and noticed the mention of VAE in the configurations. However, I couldn't find any specific guidance on where to place the VAE within the repository. Could you provide some insights or directions on this?

Thank you for your assistance!

[s9roll7]

Currently, it is not possible to read vae by itself.
The data that the checkpoint has is used.

[seemon2]

Hi, .. when running the command in the command "animatediff stylize create-config IA.mp4" (my video), encounter this error ..

Is there an issue that need resolve that I am not aware of for stylized run?

[rudekuclux]

Why my result is bad when I use openpose, I updated to the latest on the morning of September 13th I used command animatediff generate -c config/prompts/Meina.json -W 424 -H 768 -L 16 -C 16 my openpose images my result MyPrompts MyPrompts.zip

i have issue like this how do you fix it?

[s9roll7]

Why my result is bad when I use openpose, I updated to the latest on the morning of September 13th I used command animatediff generate -c config/prompts/Meina.json -W 424 -H 768 -L 16 -C 16 my openpose images my result MyPrompts MyPrompts.zip

i have issue like this how do you fix it?

#29 (comment)

[prezofpizza]

Can anyone share some prompts that have had good results? My results tend to look pretty messy.

[lyc6749]

4. controlnet_conditioning_scale

Have you solved this problem?