PaddleVideo

Introduction

PaddleVideo is a toolset for video recognition, action localization, and spatio temporal action detection tasks prepared for the industry and academia. This repository provides examples and best practice guildelines for exploring deep learning algorithm in the scene of video area. We devote to support experiments and utilities which can significantly reduce the "time to deploy". By the way, this is also a proficiency verification and implementation of the newest PaddlePaddle 2.0 in the video field.

Feature

Advanced model zoo design PaddleVideo unifies the video understanding tasks, including recogniztion, localization, spatio temporal action detection, and so on. with the clear configuration system based on IOC/DI, we design a decoupling modular and extensible framework which can easily construct a customized network by combining different modules.
Various dataset and architectures PaddleVideo supports more datasets and architectures, including Kinectics400, ucf101, YoutTube8M datasets, and video recognition models, such as TSN, TSM, SlowFast, AttentionLSTM and action localization model, like BMN.
Higher performance PaddleVideo has built-in solutions to improve accuracy on the recognition models. PP-TSM, which is based on the standard TSM, already archive the best performance in the 2D recognition network, has the same size of parameters but improve the Top1 Acc to 73.5% , and one can easily apply the soulutions on his own dataset.
Faster training strategy PaddleVideo suppors faster training strategy, it accelerates by 100% compared with the standard Slowfast version, and it only takes 10 days to train from scratch on the kinetics400 dataset.
Deployable PaddleVideo is powered by the Paddle Inference. There is no need to convert the model to ONNX format when deploying it, all you want can be found in this repository.

Overview of the kit structures

Architectures

Frameworks

Components

Data Augmentation

Recognition

TSN
TSM
SlowFast
PP-TSM
VideoTag
AttentionLSTM

Localization

Recognizer1D

Recognizer2D

Recognizer3D

Localizer

resnet
resnet_tsm
resnet_tweaks_tsm
bmn

tsm_head
tsn_head
bmn_head

Solver

Optimizer

Momentum
RMSProp

LearningRate

PiecewiseDecay

Loss

CrossEntropy
BMNLoss

Metrics

CenterCrop
MultiCrop

Batch

Mixup
Cutmix

Image

Scale
Random FLip
Jitter Scale
Crop
MultiCrop
Center Crop
MultiScaleCrop
Random Crop
PackOutput

Overview of the performance

The chart below illustrates the performance of the video recognition models both 2D and 3D architectures, including our implementation and Pytorch version. It shows the relationship between Acc Top1 and VPS on the Kinectics400 dataset. (Tested on the NVIDIA® Tesla® GPU V100.)

Note：

PP-TSM improves almost 3.5% Top1 accuracy from standard TSM.
all these models described by RED color can be obtained in the Model Zoo, and others are Pytorch results.