yael-vinker/live_sketch

large optimization/ inference time

Closed this issue · 3 comments

NippunSharma commented

Hi, this is truly a great piece of work!
One of the things I noticed is that the inference or optimization time per video generation is pretty large (I tried an example that took about 35mins). Is this expected?

rinongal commented

Hi,

35 minutes is more or less expected, yes (we reported ~30 mimutes on an A100 in the paper).

Two things you can do to reduce this:

  • If you don't need global movement you could try to run without the global path. This should cut down times in half. You may need a bit of extra parameter tuning for the local path if you do this.
  • You can try using fewer iterations. In some cases you can get decent results even with ~500 steps.
NippunSharma commented

@rinongal Just curious to know, is this large inference time due to using diffvg at each iteration for each frame?

rinongal commented

The biggest factor is the fact that we're an optimization based approach. We're using Score Distillation Sampling (SDS), and SDS often requires many steps to converge well. You can see this for example in the original DreamFusion paper that proposed SDS, which takes 1.5 hours on 4 GPUs to render a single scene.

If we could use 10 steps and not 1000 (for example, if we had better initialization or a stronger motion signal), we'd be done in 20 seconds. Hopefully some follow up works will get us there :)