LINCellularNeuroscience/VAME

Adding other data to be used in training (e.g. velocity/acceleration)

akesner1 opened this issue · 4 comments

akesner1 commented

Hi All,

Thanks to all the developers for a great tool.

I have heard & read that adding values like velocity or acceleration of the animal/body points can increase the model's performance at delineating behavioral motifs (especially for top down recordings). How should we do this? Should this be added as a column to the .csv's we put into the pose_estimation folder at the start of the pipeline? For example, if we'd want to add an overall velocity data, e.g. animal CenterPoint velocity, should we add one new column to the CSV that has this metric, or one column for X-velocity, one column for Y-velocity, and a mock column for velocity likelihood (to keep the formatting same as other metrics delivered via the DLC .csv file)? Or should this be added to one of the outputs of the initial steps in the pipeline (like after egocentric alignment)? If so, which file?

Thanks,
Drew

pavolbauer commented

Hi Drew,
Please try the following:

  1. run the pipeline including the egocentrical alignment script
  2. stack the newly added time series onto your aligned (*PE-seq-clean.npy file, https://www.nature.com/articles/s42003-022-04080-7/figures/6)
  3. increase the n_features count in you config.yaml (https://www.nature.com/articles/s42003-022-04080-7/tables/1)

Hope this helps,
Best,
Pavol

akesner1 commented

Thanks! Thats what I was thinking.

And congrats to everyone on the publication. It is a beast and lots of nice additions from the preprint!

KonradDanielewski commented

On this topic in a way. Can I change data to 3D adding z axis coordinates to the *PE-seq-clean.npy and use that?

pavolbauer commented

Hi Konrad,

Yes, you can add as many additional dimensions as you want to your PE-seq-clean.npy file.
Could be 3d coords, sensors data, ...
VAME handles that additional dims automatically.

Best
Pavol