could you please provide me the weight of VideoMAE pre-trained on Kinetics-400,I want to use the the weight to extract the features of the thumos14
Value-Jack opened this issue · 4 comments
You can use the VideoMAEModel class from Hugging Face, and set output_hidden_states=True to get the hidden states of all layers or only from the last layer.
from transformers import VideoMAEImageProcessor, VideoMAEModel
import numpy as np
import torch
processor = VideoMAEImageProcessor.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics")
model = VideoMAEModel.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics")
# Normalized video of shape (T x C x H x W).
num_frames = 16
video = list(np.random.randint(0, 256, (num_frames, 3, 224, 224)) / 255.)
pixel_values = processor(video, return_tensors="pt").pixel_values
num_patches_per_frame = (model.config.image_size // model.config.patch_size) ** 2
seq_length = (num_frames // model.config.tubelet_size) * num_patches_per_frame
bool_masked_pos = torch.randint(0, 2, (1, seq_length)).bool()
with torch.no_grad():
out = model(pixel_values, output_hidden_states=True, bool_masked_pos=bool_masked_pos)
# Output of the last layer of the model.
all_layers = out.hidden_states
# Output of each layer plus the optional initial embedding outputs.
last_layer = out.last_hidden_state
source: https://huggingface.co/docs/transformers/model_doc/videomae#videomae
Hi @Value-Jack ! You can download VideoMAE features of THUMOS, ActivityNet, HACS and FineAction from this link.
Hi @Value-Jack ! You can download VideoMAE features of THUMOS, ActivityNet, HACS and FineAction from this link.
could you please tell me why the dim of the channel of the thumos14 dataset is 1280?
I read the videoMAE model, and the shape[1] should be 768? and Do the 1280 include the flow features? or just rgb feature?
I'll appreciate it if you could answer me the questions!