Text Mining and Search project
### FOLDER'S STRUCTURE ###
Project_root
--Plots # Folder for wordclouds and models' performance plots
--Data # Data folder
----preprocessed.RData # Output df from file '2_preprocessing.R' with unique combination of matrix type and sparsity
# to be fed into '3_models.R'
----20news-bydate-test/ # Newsarticles from the test set
----20news-bydate-train/# Newsarticles from the train set
------alt.atheism/ # Topic folder
--------doc_id # Single document file referenced by id
------rec.autos/ # Topic folder (one of the 20)
------sci.crypt/ # Topic folder
--0_helperFunctions.R # Script with various functions to preprocess and to build different text representation matrixes
--helperFunctions.RData # Rdata that contains all the functions above used further on (just to simplify code estetic)
--1_exploration.R # Explore different text representations and sparsity threshold through word clouds
--2_preprocessing.R # Build dataframe with target/topic variable included, ready to be used for model training
--3_models.R # Models bulding with caret
### LIBRARIES ###
The following is a list of commands to execute in R to install all the needed packages.
install.packages( c("tm", "caret", "dplyr", "StandardizeText", "SnowballC", "wordcloud2", "tidytext", "webshot", "kernlab", "splitstackshape", "e1071", "textclean", "mgsub", "tictoc", "klaR", "promises", "C50", "inum", "NLP", "stringi", "RNewsflow", "doParallel")
webshot::install_phantomjs()