Created Date: 31 Oct 2018

Text-Pre-Processing-Text-in-Python

So are you planning to do research in the text field but not sure about how to start?
Well, why not start with pre-processing of text as it is very important while doing research in text field and its easy! while cleaning the text helps you get quality output by removing all irrelevant text and getting the forms of the words etc.
In this article, we will be covering:

Converting text to lowercase
Contraction
Sentence tokenize
Word tokenize
Spell Check
Lemmatize
Stemming
Remove Tags
Remove numbers
Remove punctuation
Remove stopwords

Let's START!
Pre-requisites:
install Python
install NLTK
pip install autocorrect

Done with the installations? okay! let's start coding!

Convert text to lower case:

Converting text to lower case as in, converting "Hello" to "hello" or "HELLO" to "hello".

Contraction

Contraction helps to expand the word form like "ain't": "am not". Contractions file has been created in my github which we will be importing to use it.

sentence tokenize

Tokenize sentences if the there are more than 1 sentence i.e breaking the sentences to list of sentence.

word tokenize

Tokenize words to get the tokens of the text i.e breaking the sentences into list of words.

Spell Check

correct the incorrect spelled words like "wrld" to "world"

Lemmatize

lemmatize the text so as to get its root form eg: functions,funtionality as function

Stemming

stemming is the process of reducing inflected (or sometimes derived) words to their word stem, base or root form

Remove Tags

Removing html tags from the text like "" using regex.

Remove Numbers

Removing numbers from the text like "1,2,3,4,5…" We usually remove numbers when we do text clustering or getting keyphrases as we numbers doesn't give much importance to get the main words. To remove numbers, you can use: .isnumeric() else .isdigit()

Remove punctuation

Removing punctuation from the text like ".?!" and also the symbols like "@#$" .

Stop words removal

Remove irrelevant words using nltk stop words like "is,the,a" etc from the sentences as they don't carry any information.

PS:

You can test all those individually from the folder "individual_python_files"
else run main.py directly
Preprocess.py is where the Preprocess class has been created for all the invidual task.

vietecon/Text-Pre-Processing-in-Python