Do you want >72% top-1 accuracy on a large video dataset? Are you tired of Kinetics videos disappearing from YouTube every day? Do you have recurring nightmares about Caffe2? Then this is the repo for you!
This is a PyTorch implementation of the Caffe2 I3D ResNet Nonlocal model from the video-nonlocal-net repo. The weights are directly ported from the caffe2 model (See checkpoints). This should be a good starting point to extract features, finetune on another dataset etc. without the hassle of dealing with Caffe2, and with all the benefits of a very carefully trained Kinetics model.
It's only a matter of time before FAIR releases a good PyTorch version of their nonlocal-net codebase, but until then, at least you have this ¯\_(ツ)_/¯
UPDATE: FAIR has released a good PyTorch video codebase. Stop using this repo. Start using that!
Amazing features:
- Only a single model (ResNet50-I3D). Parameters hardcoded with love.
- Only the evaluation script for Kinetics (training from scratch or ftuning has not been tested yet.)
- No nonlocal versions yet. One exciting NL version to choose from.
The code has been tested with Python 3.7 + PyTorch 1.1.
Pretrained Weights
Download pretrained weights for I3D and I3D-NL models from the nonlocal repo
wget https://dl.fbaipublicfiles.com/video-nonlocal/i3d_baseline_32x2_IN_pretrain_400k.pkl -P pretrained/
wget https://dl.fbaipublicfiles.com/video-nonlocal/i3d_nonlocal_32x2_IN_pretrain_400k.pkl -P pretrained/Convert these weights from caffe2 to pytorch. This is just a simple renaming of the blobs to match the pytorch model.
python -m utils.convert_weights pretrained/i3d_baseline_32x2_IN_pretrain_400k.pkl pretrained/i3d_r50_kinetics.pth
python -m utils.convert_weights pretrained/i3d_nonlocal_32x2_IN_pretrain_400k.pkl pretrained/i3d_r50_nl_kinetics.pthThe model can be created and weights loaded using
from models import resnet
net = resnet.i3_res50() # vanilla I3D ResNet50
net = resnet.i3_res50_nl() # Nonlocal versionData
Download videos using the official crawler and extract frames. This repo has a script to do this. Then create softlinks for frames and annotations:
mkdir -p data/kinetics/frames/ data/kinetics/annotations/
ln -s /path/to/kinetics/frames data/kinetics/frames/
ln -s /path/to/kinetics/annotation_csvs data/kinetics/annotations/Evaluate
Run the evaluation script to generate scores on the validation set.
# Evaluate using 3 random spatial crops per frame + 10 uniformly sampled clips per video
# Model = I3D ResNet50 Nonlocal
python eval.py --batch_size 8 --mode video --model r50_nl
# Evaluate using a single, center crop and a single, centered clip of 32 frames
# Model = I3D ResNet50
python eval.py --batch_size 8 --mode clip --model r50
# Use --parallel for multiple GPUs
python eval.py --batch_size 16 --mode clip --model r50_nl --parallel
| Model | clip (top1/top5) | video (top1/top5) |
|---|---|---|
| I3D Res50 | 0.647 / 0.853 | 0.721 / 0.902 |
| I3D Res50 NL | 0.664 / 0.864 | 0.737 / 0.912 |
You should get around 72.1% top-1 accuracy for the video using I3D Res50, and around 73.7% using the non-local version. Note that these numbers are on whatever is left of the Kinetics val set these days (~18434 videos).