dazhi654321/SLR

isolated & continuous sign language recognition using CNN+LSTM/3D CNN/GCN/Encoder-Decoder

Python

SLR

isolated & continuous sign language recognition using CNN+LSTM/3D CNN/GCN/Encoder-Decoder

Requirements

Download and extract CSL Dataset
Download and install PyTorch

Isolated Sign Language Recognition

CNN+LSTM

four layers of Conv2d + one layer of LSTM

Dataset Classes Samples Best Test Acc Best Test Loss

CSL_Isolated 100 25,000 82.08% 0.734426

CSL_Isolated 500 125,000 71.71% 1.332122
ResNet + one layer of LSTM

Dataset Classes Samples Best Test Acc Best Test Loss

CSL_Isolated 100 25,000 93.54% 0.245582

CSL_Isolated 500 125,000 83.17% 0.748759

3D CNN

three layers of Conv3d

Dataset Classes Samples Best Test Acc Best Test Loss

CSL_Isolated 100 25,000 58.86% 1.560049

CSL_Isolated 500 125,000 45.07% 2.255563

3D ResNet

Method	Dataset	Classes	Samples	Best Test Acc	Best Test Loss
ResNet18	CSL_Isolated	100	25,000	93.30%	0.246169
ResNet18	CSL_Isolated	500	125,000	79.42%	0.800490
ResNet34	CSL_Isolated	100	25,000	94.78%	0.207592
ResNet34	CSL_Isolated	500	125,000	81.61%	0.750424
ResNet50	CSL_Isolated	100	25,000	94.36%	0.232631
ResNet50	CSL_Isolated	500	125,000	83.15%	0.803212
ResNet101	CSL_Isolated	100	25,000	95.26%	0.205430
ResNet101	CSL_Isolated	500	125,000	83.18%	0.751727

ResNet (2+1)D

Dataset Classes Samples Best Test Acc Best Test Loss

CSL_Isolated 100 25,000 98.68% 0.043099

CSL_Isolated 500 125,000 94.85% 0.234880

GCN

Dataset	Classes	Samples	Best Test Acc	Best Test Loss
CSL_Skeleton	100	25,000	79.20%	0.737053
CSL_Skeleton	500	125,000	66.64%	1.165872

Skeleton+LSTM

Dataset	Classes	Samples	Best Test Acc	Best Test Loss
CSL_Skeleton	100	25,000	84.30%	0.488253
CSL_Skeleton	500	125,000	70.62%	1.078730

Continuous Sign Language Recognition

Encoder-Decoder

Encoder is ResNet18+LSTM, and Decoder is LSTM

Dataset	Sentences	Samples	Best Test Wer	Best Test Loss
CSL_Continuous	100	25,000	1.01%	0.034636
CSL_Continuous_Char	100	25,000	1.19%	0.049449

References