/nlp-for-hindi

State of the Art Language models and Classifier for Hindi language (spoken in Indian sub-continent)

Primary LanguageJupyter NotebookMIT LicenseMIT

NLP for Hindi

This repository contains State of the Art Language models and Classifier for Hindi language (spoken in Indian sub-continent).

The models trained here have been used in Natural Language Toolkit for Indic Languages (iNLTK)

Dataset

Created as part of this project

  1. Hindi Wikipedia Articles - 172k

  2. Hindi Wikipedia Articles - 55k

  3. Hindi Movie Reviews Dataset

  4. Hindi Text Short Summarization Corpus

  5. Hindi Text Short and Large Summarization Corpus

Open Source Datasets

  1. BBC News Articles : Sentiment analysis corpus for Hindi documents extracted from BBC news website.

  2. IIT Patna Product Reviews : Sentiment analysis corpus for product reviews posted in Hindi.

  3. IIT Patna Movie Reviews : Sentiment analysis corpus for movie reviews posted in Hindi.

Results

Language Model Perplexity (on validation set)

Architecture/Dataset Hindi Wikipedia Articles - 172k Hindi Wikipedia Articles - 55k
ULMFiT 34.06 35.87
TransformerXL 26.09 34.78

Note: Nirant has done previous SOTA work with Hindi Language Model and achieved perplexity of ~46. The scores above aren't directly comparable with his score because his train and validation set were different and they aren't available for reproducibility

Classification Metrics

ULMFiT
Dataset Accuracy MCC Notebook to Reproduce results
BBC News Articles 78.75 71.61 Link
IIT Patna Movie Reviews 57.74 37.23 Link
IIT Patna Product Reviews 75.71 59.76 Link

Visualizations

Word Embeddings
Architecture Visualization
ULMFiT Embeddings projection
TransformerXL Embeddings projection
Sentence Embeddings
Architecture Visualization
ULMFiT Encodings projection

Results of using Transfer Learning + Data Augmentation from iNLTK

On using complete training set (with Transfer learning)
Dataset Dataset size (train, valid, test) Accuracy MCC Notebook to Reproduce results
IIT Patna Movie Reviews (2480, 310, 310) 57.74 37.23 Link
On using 20% of training set (with Transfer learning)
Dataset Dataset size (train, valid, test) Accuracy MCC Notebook to Reproduce results
IIT Patna Movie Reviews (496, 310, 310) 47.74 20.50 Link
On using 20% of training set (with Transfer learning + Data Augmentation)
Dataset Dataset size (train, valid, test) Accuracy MCC Notebook to Reproduce results
IIT Patna Movie Reviews (496, 310, 310) 56.13 34.39 Link

Pretrained Models

Language Models

Download pretrained Language Models of ULMFiT, TransformerXL trained on Hindi Wikipedia Articles - 172k and Hindi Wikipedia Articles - 55k from here

Tokenizer

Unsupervised training using Google's sentencepiece

Download the trained model and vocabulary from here