/PSL-GAN

Generative Adversarial Graph Convolutional Networks for Peruvian Sign Language Synthesis

Primary LanguageJupyter NotebookMIT LicenseMIT

PSL-GAN

Generative Adversarial Graph Convolutional Networks for Peruvian Sign Language Synthesis.

System requirements

  • Python 3.8.11

  • Create a new environment using conda create -y -n psl-gan python=3.8.11

  • N_CUDA environment variable is defined to choose the GPU in case of having more than one

    pip3 install -r requirements.txt

Dataset

This repository contains the implementantion of Kinetic GAN applied to Peruvian Sign Language. It is used as a data augmentation method.

The Peruvian Sign Language dataset was created by https://github.com/gissemari/PeruvianSignLanguage. This folder contains the following data:

  • segmented_signs.zip: zip file with the segmented signs in MP4 format
  • raw_data_cocopose.json: located in zip file. Contains the landmarks got by COCO-WholeBody pose estimation model
  • raw_data_mediapipe.json: Contains the landmarks got by MediaPipe pose estimation model (body and hands)

To generate the pickle file with the shape of N x C x T x V, where N is the number of samples, C the number of coordinates, T the number of frames, and V the number of joints, run the following code

  • 27 number of joints
python3 create_dataset.py --minframes <MIN_FRAMES> --mininstances <MIN_FRAMES>  --leftHandLandmarks --rightHandLandmarks --rawCocoDataset <PATH_JSON_COCOPOSE> --inputPath <PATH_SEGMENTED_SIGNS> --nLandmarks 27 --useCoco --rawDataset <PATH_JSON_MEDIAPIPE>
  • 24 number of joints
python3 create_dataset.py --minframes <MIN_FRAMES> --mininstances <MIN_FRAMES>  --leftHandLandmarks --rightHandLandmarks --rawCocoDataset <PATH_JSON_COCOPOSE> --inputPath <PATH_SEGMENTED_SIGNS> --nLandmarks 24 --useCoco --rawDataset <PATH_JSON_MEDIAPIPE> --addExtraJoint

Training the Neural Network

  1. Run the following code for training without wandb tracking.
python3 train.py