[Test] Stylize Video (with mask)
s9roll7 opened this issue · 16 comments
- sample3
checkpoint : mistoonAnime_v20.safetensors
motion_module : mm_sd_v15_v2.ckpt
steps : 20
guidance_scale : 10
[0]
ip_adapter_plus ("is_plus_face": false, "is_plus": true) / scale : 0.5
controlnet_openpose / controlnet_conditioning_scale : 1.0
controlnet_normalbae / controlnet_conditioning_scale : 0.5
[1]
ip_adapter_plus / scale : 0.5
controlnet_tile / controlnet_conditioning_scale : 1.0
mask_sample3.mp4
- sample4
checkpoint : mistoonAnime_v20.safetensors
motion_module : mm_sd_v15_v2.ckpt
steps : 20
guidance_scale : 10
[0]
ip_adapter_plus ("is_plus_face": false, "is_plus": true) / scale : 0.5
controlnet_openpose / controlnet_conditioning_scale : 1.0
controlnet_ip2p/ controlnet_conditioning_scale : 0.5
controlnet_normalbae / controlnet_conditioning_scale : 0.25
[1]
ip_adapter_plus / scale : 0.5
controlnet_tile / controlnet_conditioning_scale : 1.0
mask_sample4.mp4
- sample5
checkpoint : mistoonAnime_v20.safetensors
motion_module : mm_sd_v15_v2.ckpt
steps : 20
guidance_scale : 10
[0]
ip_adapter_plus ("is_plus_face": false, "is_plus": true) / scale : 0.5
controlnet_openpose / controlnet_conditioning_scale : 1.0
controlnet_normalbae / controlnet_conditioning_scale : 0.25
[1]
ip_adapter_plus / scale : 0.5
controlnet_tile / controlnet_conditioning_scale : 1.0
mask_sample5.mp4
is there a way to directly mask the latents instead of post-processing, i think it would lead to much better results @s9roll7
Do you mean inpaint?
That sounds interesting too.
yes, with inpainting it's much better to only generate the masked area without the need to generate the whole frame and cut out the foreground with masks which leads to unnecessary detail being generated like in the examples above, i'm trying to implement it for this repo but i'm still facing some issues, i'd love to talk to you about it @s9roll7.
I think each method has its advantages.
The advantage of the method I implemented this time is that we can change the shape of the conversion target
hi @s9roll7 the videos are said to be corrupted. Mind re uploading the video ? thanks
hi @s9roll7 the videos are said to be corrupted. Mind re uploading the video ? thanks
I have no problems in my environment.
you right. works on another one's computer but not mine's 2 browser. Thanks anyway!
HELLO! I have some question about Stylize Video (with mask)
- When I generate the background images, the image size changed (I saw Prompt changes also) is it correct or I need to modify by my selg?
- can I just change background image without changing front ground? how should I setup?
thanks!
- When I generate the background images, the image size changed (I saw Prompt changes also) is it correct or I need to modify by my selg?
I'm a little unclear on the situation.
Do you mean that some areas of the picture are missing?
Can you upload the image that went wrong?
- can I just change background image without changing front ground? how should I setup?
animatediff stylize generate BG_STYLYZE_DIR
"composite": {
"fg_list": [
{
"path": "FG_STYLYZE_DIR/00_controlnet_image/controlnet_tile",
"mask_path": "FG_STYLYZE_DIR/00_mask",
"mask_prompt": "person"
},
{
"path": " absolute path to frame dir ",
"mask_path": " absolute path to mask dir (this is optional) ",
"mask_prompt": "cat"
}
],
"bg_frame_dir": "BG_STYLYZE_DIR/time_stamp_str/00-341774366206100",
"hint": ""
},
I'm a little unclear on the situation.
Do you mean that some areas of the picture are missing?
Can you upload the image that went wrong?
My orgional sitting for prompt.json:
"width": 512,
"height": 336,
"length": 180,
"context": 16,
"overlap": 4,
"stride": 0
and the setting in bg_2023-10-19T19-28-01 become
"width": 448,
"height": 568,
"length": 180,
"context": 16,
"overlap": 4,
"stride": 0
which the width and high changed, I thought the fd may change the width and height but not sure if bg will change or not?
I see
I'm resizing it by default to save time.
If you want to improve quality, change to a larger size.
I see I'm resizing it by default to save time. If you want to improve quality, change to a larger size.
Since the aspect ratio of the background does not change, there may be no point in adjusting the size.
I will change the background size so that it does not change in the next update
HI, @s9roll7 ,
After applying the "stylize composite" effect, the result appears green. Should I adjust some parameters to control the amount of green?
Origin Image from result
cp_2023-10-20T08-49-58/fg_00_2023-10-20T08-49-58/00000000.png
cp_2023-10-20T08-49-58/bg_00_2023-10-20T08-49-58/00000000.png
I think it is the influence of the composition method.
The green color of its legs comes from the color of the trees in the background.
@s9roll7 Is there a way to minimize the environmental impact?
I tried modifying the compose method with a simplified OpenCV version (code bellow). The color of the leg looks good, but the edges of the mask appear fragile.
I need to adjust the LaplacianPyramidBlender, but which parameters can I set?
Is there a better way to enhance composite color and mask edge quality?
Thank you very much.
bg = Image.open(bg)
fg = Image.fromarray(fg_array)
# Create mask
fg_pixels = fg.load()
width, height = fg.size
mask = Image.new('L', fg.size)
mask_pixels = mask.load()
for y in range(height):
for x in range(width):
r, g, b = fg_pixels[x, y]
if g > r and g > b: # Simple green detection
mask_pixels[x, y] = 0
else:
mask_pixels[x, y] = 255
# Blend images
result = Image.composite(fg, bg, mask)
# Save result
result.save(save_path)
I'm a little unclear on the situation.
Do you mean that some areas of the picture are missing?
Can you upload the image that went wrong?
My orgional sitting for prompt.json:
"width": 512,
"height": 336,
"length": 180,
"context": 16,
"overlap": 4,
"stride": 0and the setting in bg_2023-10-19T19-28-01 become "width": 448, "height": 568, "length": 180, "context": 16, "overlap": 4, "stride": 0 which the width and high changed, I thought the fd may change the width and height but not sure if bg will change or not?
@s9roll7 Is there a way to minimize the environmental impact?