- Intro: Project of automatically finding helpful reviews.
Using text mining technique. Include natural language processing, sentiment analysis and machine learning technique.
Language: Python. (standard library: xlrd, jieba, gensim, nltk, scikit-learn)
This project include four main parts:
-
Review data set
-
Text preprocessing module.
-
Review helpfulness feature extraction module.
-
Review helpfulness prediction module.
-
Review data set
This dataset include review data of 7 cellphone brands grabbed from (http://mobile.zol.com.cn).
- Text preprocessing module
Functions include: Read txt file and excel file, doing Chinese word segmentation and postag.
Using Python xlrd library to read excel data.
Using Python jieba library to do Chinese word segmentation and postag
Files include:
User dictionary to increase segmentation accuracy.
Stopwords and sentiment stopwords to filter irrelevant words.
- Feature extraction module
There is four categories of review helpfulness feature.
- Linguistic features
6 features including review words and sentences number, review average length, review adjectives, adverbs and verbs number. - Informative features
4 features including product name, brand and attributes' appearing times in a review. And review centroid score. - Difference features
2 features including review entropy and perplexity socre. - Sentiment feautres
8 features including review positive/negative score, average score and standard deviation score. And review positive/negative probability score.
Using Python nltk library to compute review entropy/perplexity and doing sentiment analysis.
Using Python gensim library to calculate review words tf-idf weight.
Files include:
Labeled positive and negative review corpus.
Trained sentiment classifier, aim to automatically classify review positive and negative.
Sentiment dictionary including positive and negative words and adverbs of degree.
Raw sentiment dictionary including Hownet and NTUSD.
- Review helpfulness prediction module
This module use features calculated from above module as training set. Using machine learning method to train review helpfulness classifier. We test five popular machine learning algorithm and use cross validation method to evaluate helpfulness prediction accuracy.
Using Python scikit-learn library to train classifier and evaluated classifier performance.
Files include:
Feature vector matrix with different threshold.