/BARD-Bangla-Article-Classifier

Primary LanguagePython

BARD Bangla Article Classifier

Video Demo: bit.ly/BARD_VIDEO_DEMO
Dataset Link: bit.ly/BARD_DATASET

Dataset Details

Category No. of Documents No. of Words Average Sentences per Document Average words per Sentence
State 242860 57019465 18.50 13.356
Economy 18982 4915141 20.18 13.378
International 32203 7096111 18.47 12.493
Entertainment 31293 6706563 21.70 10.236
Sports 50888 12397415 22.80 11.069

Abstract

In the literature article classification is well studied, where several supervised leaning models have been proposed by utilizing large textual data corpus. Despite several comprehensive textual dataset available for different language, only a few small datasets available for Bangla articles. As a result, a couple works address the Bangla document classification and could not able to learn sophisticated supervised learning model. In this work, we curated a large dataset of Bangla article which help us to train several supervised learning model using some sophisticate features, such as word2vec. As word2vec preserves semantic features, it shows superior performance in text classification. Moreover, Neural Network with word2vec features shows promising performance compared to other state-of-the-art-work in the text classification.
Furthermore, we deployed our proposed Bangla content classifier as a web application, which is accessible here. and the video demo of this application is available here. Additionally, we open-sourced the BARD dataset and source code of this work in this repository.

Statistical Analysis

Figure 01: Most frequent words in each categories.

We performed textual statistical analysis on the BARDdataset articles and results are presented in Fig. 1 and 2. Thefrequency distributions of the top 20 most frequent words foreach of the five categories are depicted in Fig. 1. From thisanalysis, we can easily identify that all the categories havethe similar most frequent words. In other words, these frequentwords do not help to categorize the articles. Hence, we removearound 25 most frequent from all the articles and performedthe statistical analysis again on the filtered dataset, which ispresented in Fig. 2. Now, this filtered frequency distributiondepicted that each category has some unique distribution ofwords, which may contribute to categorize the articles.

Figure 02: Most frequent words in each categories after removing stop words.