facebookresearch/co-tracker

How to run the online tracker for every frame?

mizeller opened this issue · 4 comments

mizeller commented

In the online demo, the tracker runs on every 4th step because of this if-clause in online_demo.py:

if i % model.step == 0 and i != 0:
       _process_step(...)

Is it possible to run the tracker on every frame somehow?
I tried adjusting the model.window_len which ultimately determines model.step, but could not figure out a working solution somehow.

nikitakaraevv commented

Hi @mizeller, yes, the current online model indeed only works with 4-frame windows. There is going to be a lag of 3 frames, and we don't have a solution for this yet.

  • 👍1
mizeller commented

@nikitakaraevv Thank you for the speedy reply! I guess I'll continue with the regular CoTrackerPredictor if there is no easy "hacky" solution in that case :-)

mizeller commented

(BTW: the variables W and H are not initialised in the case of is_first_step=True and queries is not None in the online predictor --> small bug)

if is_first_step:
      self.model.init_video_online_processing()
      if queries is not None:
          B, N, D = queries.shape
          assert D == 3
          queries = queries.clone()
          queries[:, :, 1:] *= queries.new_tensor(
          [
                (self.interp_shape[1] - 1) / (W - 1), # here
                (self.interp_shape[0] - 1) / (H - 1), # and here
                ....
nikitakaraevv commented

Thanks! Fixed it: fac2798

  • 👍1