Reproducing 3.6 HD model
Closed this issue · 5 comments
Hello! I am trying to reproduce the 3.6 model linked in the Readme.md
However, the training procedure in RIFE_HDv3.py file shipped with the pretrained model appears to be broken
`if training:
self.optimG.zero_grad()
loss_G = loss_cons + loss_smooth * 0.1
loss_G.backward()
self.optimG.step()
else:
flow_teacher = flow[2]
return merged[2], {
'mask': mask,
'flow': flow[2][:, :2],
'loss_l1': loss_l1,
'loss_cons': loss_cons,
'loss_smooth': loss_smooth,
}`
The loss_cons variable is not assigned anywhere, also L1 loss is calculated but not used
What loss combination should i use to reproduce v3.6?
Also what file should i use for training, RIFE_HDv3.py shipped with the model or RIFE.py presented in the repo?
Also, was that model also trained just on Vimeo triplet dataset?
https://drive.google.com/file/d/1zoSz7b8c6kUsnd4gYZ_6TrKxa7ghHJWW/view?usp=sharing It's my whole training repo.
vimeo septuplet dataset is used.
https://drive.google.com/file/d/1zoSz7b8c6kUsnd4gYZ_6TrKxa7ghHJWW/view?usp=sharing It's my whole training repo. vimeo septuplet dataset is used.
Thank you for sharing!
but i still have some questions
- Which model among those posted in the repo will it reproduce?
- All pretrained models posted in the repo are called HD models. But vimeo triplet and septuplet datasets are not HD. Am I missing something?
- What exactly is the nori2 module used in your code?
- Your dataset.py code seems to rely on your local files:
read('meta.01557630')
read('meta.015577c0')
read('meta.015579f5')
read('meta.015580f0')
read('meta.0155852c')
read('meta.01558630')
with nori.smart_open('s3://chenmingrui/datasets/vimeo_septuplet/vimeo_septuplet.json') as f:
for data in json.load(f):
self.train_data.append((0, data))
how should I use it?
Hi, 'HD model' just means that we did some training optimization on the resolution, the data set was not changed.
There are some internal distributed data providing libraries here. You may need to refer to https://github.com/megvii-research/ECCV2022-RIFE/blob/main/dataset.py for modifications. The 'read' function was originally used to read pre-processed optical flow labels and has been deprecated in training.
I'm sorry that I don't have time to put together a repo so that everyone can reproduce "practical RIFE". My code can only provide some detailed implementation reference.
Understood. Thank you for cooperation!
Do I understand correctly that septuplet set was used exactly for arbitrary timestamp training? and the pretrained 3.6 model and the model you provided in that training code are indeed arbitrary timestamp models?
Exactly.