/stopwords

Primary LanguageJupyter Notebook

stopwords

Code for the paper:

M. Gerlach, H. Shi, L.A.N. Amaral: "A universal information theoretic approach to the identification of stopwords" (2019)

Structure

  • data.
    Contains the different language and gene datasets
  • code
    • cluster_output
      Output files from running bulk analysis of topic models and document classification in cluster_scripts/
    • cluster_scripts
      Scripts run on Quest to do bulk analysis for topic models and document classification. submitted *_jobs.py. requires running the stopword-statistics beforehand (see scripts/). also requires installation of the different topic-models (see below).
    • figures
      figures from final analysis. run notebooks in figures_notebooks/
    • figures_notebooks create the final figures contained in the manuscript.
    • output
      Output files from running the stopword statistics in scripts/
    • src
      internal source-files
    • src_tm
      external source-files from topic models used in the analysis
    • tmp
      temporary folder needed to store intermediate results when running topic models.

installation steps

  • install required packages
    • python (3.6.3)
    • gensim (0.13.1) for ldavb topic model
    • pandas (0.24.2)
    • jupyter (1.0.0)
    • scikit-learn (0.21.2)
  • configure the topic models (code/src_tmp)
    • mallet
      $ cd code/src_tm/external/mallet-2.0.8RC3
      $ ant
    • hdp
      cd code/src_tm/external/hdp-bleilab/hdp-faster
      make

running

  • Calculating stopword statistics of each corpus
    run the files in code/scripts/script_run-stopword-statistics_*.py. this will save statistics on stopwords (I, tfidf, ...) for each corpus in code/output
  • Run topic models and document classification
    I submitted the jobs running the bulk analysis using the scripts in code/cluster_scripts/*_jobs.py to the cluster; this likely needs to be adapted if you want to run it somewhere else. The corresponding code is in code/cluster_scripts/*_code.py. The results are saved in code/cluster_output.
  • make the figures
    The notebooks in code/figures_notebooks/*.ipynb use the results from the previous steps to generate the figures. they will be saved in code/figures/