Questions about implementation

[yjhong89]

Hi! Thanks for sharing training code!

While I analyzing implementation in details and have few questions.

1. Pyramid-Flow/pyramid_dit/pyramid_dit_for_video_gen_pipeline.py

   Line 451 in e4b02ef

   clean_latent = latents_list[i_s][index::column_size] # [bs, c, t, h, w]

• Why use indexing [index::column_size] ? Since latent_list[i_s] would have shape of [bs, c, t, h, w] so
  latents_list[i_s][index::column_size] means just getting one batch, isn't it?

2. How video sync group works?

• If I use 8 gpus and default parallel group hyper-parameter setting, sp_group_size and video_sync_group would be 8.
• Since sequential parallel already split long tokens so every gpu gets access to the same video input, why video_sync_group is necessary??

3. When extract video latent in advance, all videos have same fps ?? Since this line means if "frame" is not specified in annotation, extract first 121 frames

• Pyramid-Flow/tools/extract_video_vae_latents.py

  Line 118 in e4b02ef

  frame_indexs = video_item['frames'] if 'frames' in video_item else list(range(self.num_frames))

4. Why multiplying 2 in here? to preserve variance for each stage ?

• Pyramid-Flow/pyramid_dit/pyramid_dit_for_video_gen_pipeline.py

  Line 424 in e4b02ef

  cur_noise = F.interpolate(cur_noise, size=(height, width), mode='bilinear') * 2

Thanks!

[jy0205]

Here are the answers to your questions:

1. latents_list[i_s][index::column_size] aims to get a batch of samples that belong to the same stage.
2. We do not use the sequence parallel in the training. The code about sequence parallel is for the multi-gpu inference. The param video_sync_group is for controlling the group of processes that accept the same input sample.
3. We directly use 24 fps for training. The frames key means you can specify the frame indexes you want to extract.
4. Multiplying is to make the variance of noise still equal to 1 after bilinear interpolation. (Statisfy standard Gaussian)

[yjhong89]

Thanks!

[yjhong89]

Another questions

• Supports I2V training ? Current implementations seems only support t2v.

[feifeiobama]

Theoretically, it naturally performs I2V training during autoregressive training (since the first frame is an image). However, we have not explicitly optimized for I2V, so the performance may be suboptimal. We are working on some improvements and will share them in due time.

[yjhong89]

Yes, sounds right. autoregressive training naturally doing I2V training.

• Looking forward to share! Thanks.

Another question ?

• Then, why scale factor for first frame and remaining frames is different ?? (using same vae)

• Pyramid-Flow/pyramid_dit/pyramid_dit_for_video_gen_pipeline.py

  Line 582 in e4b02ef

  # is video

[feifeiobama]

Great observation! Please refer to #28 (comment).

[yjhong89]

Thanks for quick answer!