The problem with rabbit-jump-p2p.yaml

[jiajiaxiaoskx]

Hi, I have a problem with rabbit-jump-p2p.yaml.
After I train the model using rabbit-jump-tune.yaml, the fine-tuned checkpoint is stored in the output folder, and when I use rabbit-jump-tune.yaml to edit the video, what the road should I use in pretrained_model_path config(line 1), since there are two folders (stable-diffusion-v1.5 and output) I have to load the model.
Thanks for answering!

[ShaoTengLiu]

You need to use rabbit-jump-p2p.yaml to edit. Thanks.

[jiajiaxiaoskx]

Sorry for the mistake I made. I have another error for run_videop2p.py following your instruction

DDIM inversion...
Null-text optimization...
Start Video-P2P!
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:32<00:00, 1.84s/it]
run_videop2p.py:652: RuntimeWarning: invalid value encountered in cast
inversion.append( Image.fromarray((sequence1[i] * 255).numpy().astype(np.uint8)) )
run_videop2p.py:653: RuntimeWarning: invalid value encountered in cast
videop2p.append( Image.fromarray((sequence2[i] * 255).numpy().astype(np.uint8)) )

How can I fix this problem?
Thanks a lot!

[ShaoTengLiu]

Hi, in your logs, I only see some warnings. Can you clarify your problem and show me your running script?

[jiajiaxiaoskx]

It seems that the problem may lie in run_videop2p.py in the

with torch.no_grad():
--> sequence = ldm_stable(
prompts,
generator=generator,
latents=x_t,
uncond_embeddings_pre=uncond_embeddings,
controller = controller,
video_length=video_len,
fast=fast,
).videos
sequence1 = rearrange(sequence[0], "c t h w -> t h w c")
sequence2 = rearrange(sequence[1], "c t h w -> t h w c")
inversion = []
videop2p = []

the pixel values in sequence are all nan, making it impossible to generate the accurate gif file.

the config file I used is as follows:

pretrained_model_path: "./outputs/rabbit-jump"
image_path: "./data/rabbit"
prompt: "a rabbit is jumping on the grass"
prompts:

• "a rabbit is jumping on the grass"
• "a origami rabbit is jumping on the grass"
  blend_word:
• 'rabbit'
• 'rabbit'
  eq_params:
  words:
  • "origami"
    values:
  • 2
    save_name: "origami"
    is_word_swap: False

Besides, when training, the step loss becomes nan after ~200 steps, and the output pixel values of validation_pipeline( line 332 in run_tuning.py) are also nan, I think there must be something wrong!
Thanks a lot!

[ShaoTengLiu]

Dose
python run_videop2p.py --config="configs/rabbit-jump-p2p.yaml" --fast
work for you?

[ShaoTengLiu]

I will temporally close this issue. You are welcome to reopen it if you still have this problem.