We use pdfplumber for read the 10 pdfs examples files from Panama National Assembly website.
In this Jupyter Notebook we are testing if the data can works properly with NLP, thats why we are using only one PDF located in resources/pdfs. We crop the footer and index number then use Regex for match only the paragraphs and not the name of the person who said it and save the final data in a txt call "Resultado.txt". After the data extract we use Spacy for parse, tokenization, NER, Similarity, vectorization and SentimentClassifier with nltk (Not so good).
We only using 10 pdfs, extract the data, clean the data with regex and then pass to a txt file. After the data is ready, we use nltk and gensim for make three Topic Models (HDP, LSI and LDA). After the models are ready we compare in terms of coherence and the winner was HDP but the data need to be improve (Read recommendations).
The data ready for NLPTesting.
There are the ten PDFS using for Testing_TopicModeling.
There are the ten txt files ready for the Topic Modeling.
Only one PDF used in NLPTesting.
Txt files NOT ready, only the exctract text without regex and crop.
- Each PDF can have many topics, depending on the debate or law that is being discussed, the text needs to be divided according to the law or debate to obtain better results.
- Add more some words in the black list for avoid unimportant words.