I plotted label vs frequency graphs and most frequent words.
We can see that the data is very imbalanced.
-
Orthographic Vector:
It basically checks on each position of the string, what character is present.
'x' - > ASCII characters,
'c' -> Small letters,
'C' -> Capital letters,
'n' -> Numbers,
'p' -> Punctuations
For example [['Hi, my name is Devashish'], ['How are you?']] will return [['CcpxccxccccxccxCcccccccc'], ['Cccxcccxcccp']]
Then after getting the mapping, it creates a vector for it. [[2, 1, 4, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 2, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 4]]
'x' -> 0,
'c' -> 1,
'C' -> 2,
'n' -> 3,
'p' -> 4,
The max length of this vector is 20 and it replaces characters from ortho mapping with numbers on that position of the string. -
**Pos Tag: **
Position of Speech Tags.
For this I am using nltk.pos_tag.
Here are the definitions of each tags:
CC -> coordinating conjunction
CD -> cardinal digit
DT -> determiner
EX -> existential there (like: “there is” … think of it like “there exists”)
FW -> foreign word
IN -> preposition/subordinating conjunction
JJ -> adjective ‘big’
JJR -> adjective, comparative ‘bigger’
JJS -> adjective, superlative ‘biggest’
LS -> list marker 1)
MD -> modal could, will
NN -> noun, singular ‘desk’
NNS -> noun plural ‘desks’
NNP -> proper noun, singular ‘Harrison’
NNPS -> proper noun, plural ‘Americans’
PDT -> predeterminer ‘all the kids’
POS -> possessive ending parent’s
PRP -> personal pronoun I, he, she
PRP -> possessive pronoun my, his, hers
RB -> adverb very, silently,
RBR -> adverb, comparative better
RBS -> adverb, superlative best
RP -> particle give up TO, to go ‘to’ the store.
UH -> interjection, errrrrrrrm
VB -> verb, base form take
VBD -> verb, past tense took
VBG -> verb, gerund/present participle taking
VBN -> verb, past participle taken
VBP -> verb, sing. present, non-3d take
VBZ -> verb, 3rd person sing. present takes
WDT -> wh-determiner which
WP -> possessive wh-pronoun whose
WRB -> wh-abverb where, when
- Method 1: Normal Classification:
I am using support vector classification for this.
Evaluation: Recall is very low and we have many false negatives - Method 2: Downsampled Classification:
Since we are dealing with imbalanced data, I have downsampled the class with label('O') and am using support vector classification for this.
Evaluation: Accuracy is very less. - Method 3: Upsampled Classification:
Since we are dealing with imbalanced data, I have upsampled the minority and am using naive bayes classification because the number of samples increases for this.
Evaluation: Accuracy is very less. - Method 4: Penalized Classification:
Since we are dealing with imbalanced data, I tried penalizing the training model. I used SVC for this with class_weight='balanced'. But it took forever to run so I decided to skip this.
Evaluation: Too much time. - Method 5: Random Forest:
Since the data in imbalanced, random forest is a good algorithm for this.
Evaluation: It performed better than all the other models till now. It was able to classify other labels also. - Method 6: Reading text as tweets:
For this, I refered to http://noisy-text.github.io/2017/pdf/WNUT19.pdf.
I have implemented a multi task neural network that aims at generating named entities in user generated text.
The model captures some orthographic features at the character level by using convulation neural network, PoS features by using Bidirectional Long-Short Term Memory.Once the network is trained, I used it as a feature extractor to feed a Conditional Random Fields (CRF) classifier.
For example: Lets consider the two texts 'Hi Paris' and 'Welcome to Paris'.
Here in the first text, 'Paris' can be reffered as Name entity and in the second text, 'Paris' can be reffered as Location entitity.
This gave me an idea that the outputs from the previous word/layer makes difference in identifying the entity.
Evaluation: It performed decent and as it made the most sense, I decided to go with this.
For evaluation, since this is very imbalanced data, I have used confusion matrix to check false negatives. I have also used accuracy for some cases.
**All the models had many false negatives. For future improvement, I should extract some more features. **
For bonus, I clustered the predicted labels based on their probabilities. I used KMeans clustering.
Top likely transitions:
B-person -> I-person 3.936851
B-location -> I-location 2.291192
I-product -> I-product 1.705511
O -> O 1.632621
B-creative-work -> I-creative-work 1.417550
I-creative-work -> I-creative-work 1.386468
B-group -> I-group 1.333125
B-product -> I-product 1.057185
I-location -> I-location 0.951327
I-group -> I-group 0.726022
O -> B-person 0.531396
B-corporation -> I-corporation 0.516798
O -> B-location 0.494253
O -> B-group 0.371175
O -> B-corporation 0.290107
Top unlikely transitions:
B-location -> B-person -0.140064
B-person -> B-location -0.142025
B-group -> O -0.152444
B-person -> B-person -0.153057
I-group -> O -0.157578
I-creative-work -> O -0.238135
I-product -> O -0.587993
B-product -> O -0.602447
B-creative-work -> O -1.781312
O -> I-corporation -2.033485
O -> I-group -2.725038
O -> I-product -3.081457
O -> I-creative-work -3.115185
O -> I-location -3.260909
O -> I-person -3.946296