taoyang1122/adapt-image-models

why the second dimension is n*b?

Closed this issue · 4 comments

007invictus commented

adapt-image-models/mmaction/models/backbones/vit_clip.py

Line 80 in 4da311e

xt = rearrange(x, 'n (b t) d -> t (b n) d', t=self.num_frames)

taoyang1122 commented

Hi, it is combining the spatial dimension with the batchsize dimension, and do self-attention on the temporal dimension in the following self-attention layer.

007invictus commented

why not (b n)* t*d?

taoyang1122 commented

because the self-attention is applie to the first dimension.

  • 👍3
007invictus commented

Thank you for your reply! It turns out I was careless in looking up the API definition.

  • 👍1