clips/gsoc2018

Is NER in anonymizer working ?

lordzuko opened this issue · 1 comments

Hi,

I was testing the text anonymizer, but I can't find the anything being anonymized, I figure you must must be identifying that using NER, and then mark them something else. Correct me if I am wrong:

Input:

Mohandas Karamchand Gandhi (/ˈɡɑːndi, ˈɡændi/;[3][needs Gujarati IPA] Hindustani: [ˈmoːɦəndaːs ˈkərəmtʃənd ˈɡaːndʱi] (About this sound listen); 2 October 1869 – 30 January 1948) was an Indian activist who was the leader of the Indian independence movement against British rule. Employing nonviolent civil disobedience, Gandhi led India to independence and inspired movements for civil rights and freedom across the world. The honorific Mahātmā (Sanskrit: "high-souled", "venerable")[4] – applied to him first in 1914 in South Africa[5] – is now used worldwide. In India, he is also called Bapu (Gujarati: endearment for father,[6] papa)[6][7] and Gandhi ji, and known as the Father of the Nation.[8][9]

Output:

Mohandas Karamchand Gandhi /ˈɡɑːndi , ˈɡændi/;[3][needs Gujarati IPA Hindustani : [ ˈmoːɦəndaːs ˈkərəmtʃənd ˈɡaːndʱi ] ( About this sound listen ) ; 2 October 1869 – 30 January 1948 was an Indian activist who was the leader of the Indian independence movement against British rule . Employing nonviolent civil disobedience , Gandhi led India to independence and inspired movements for civil rights and freedom across the world . The honorific Mahātmā ( Sanskrit : " high - souled " , " venerable")[4 ] – applied to him first in 1914 in South Africa[5 ] – is now used worldwide . In India , he is also called Bapu ( Gujarati : endearment for father,[6 ] papa)[6][7 ] and Gandhi ji , and known as the Father of the Nation.[8][9 ]

Hey,

To use the NER system, one needs to do two things.

  1. Add the attribute and select the appropriate action(Deletion, Generalization or Suppression)
  2. Link an Alias to the attribute. The alias is the Tag that the NER system provides. In this case, since we are using Spacy, the alias you should add is PERSON. Add/Link the alias with the "Attribute".

Your system should now work. Hope this helps!