goru001/nlp-for-gujarati

State of the Art Language models and Classifier for Gujarati, which is a language native to the Indian state of Gujarat

NLP for Gujarati

This repository contains State of the Art Language models and Classifier for Gujarati, which is a language native to the Indian state of Gujarat.

The models trained here have been used in Natural Language Toolkit for Indic Languages (iNLTK)

Dataset

Created as part of this project

1. Gujarati Wikipedia Articles

2. Gujarati News Dataset

Open Source Datasets

1. iNLTK Headlines Corpus - Gujarati : Uses the Gujarati News Dataset prepared above.

Results

Language Model Perplexity (on validation set)

Architecture/Dataset	Gujarati Wikipedia Articles
ULMFiT	34.12
TransformerXL	28.12

Classification Metrics

ULMFiT

Dataset	Accuracy	MCC	Notebook to Reproduce results
iNLTK Headlines Corpus - Gujarati	91.05	86.09	Link

Visualizations

Word Embeddings

Architecture	Visualization
ULMFiT	Embeddings projection
TransformerXL	Embeddings projection

Results of using Transfer Learning + Data Augmentation from iNLTK

On using complete training set (with Transfer learning)

Dataset	Dataset size (train, valid, test)	Accuracy	MCC	Notebook to Reproduce results
iNLTK Headlines Corpus - Gujarati	(5269, 659, 659)	91.05	86.09	Link

On using 10% of training set (with Transfer learning)

Dataset	Dataset size (train, valid, test)	Accuracy	MCC	Notebook to Reproduce results
iNLTK Headlines Corpus - Gujarati	(526, 659, 659)	80.88	70.18	Link

On using 10% of training set (with Transfer learning + Data Augmentation)

Dataset	Dataset size (train, valid, test)	Accuracy	MCC	Notebook to Reproduce results
iNLTK Headlines Corpus - Gujarati	(526, 659, 659)	81.03	70.44	Link

Pretrained Models

Language Models

Download pretrained Language Models from here

Tokenizer

Trained tokenizer using Google's sentencepiece

Download the trained model and vocabulary from here