/StackExchange-Miner

Analyse data and study interesting insights of various stack exchange forums, ranging from hinduism to data science, using their data dumps.

Primary LanguagePython

StackExchange-Miner

Requirements

Hardware:

  • Atleast 8GB of RAM
  • Internet Connectivity

Software:

  • OS: Ubuntu

Installation Instructions

Following Frameworks are needed to run the project:

  • NodeJS
  • Python3
  • R

Please follow the official instructions to install them properly. Make sure that the following commands get executed without any errors:

node --version
npm --version
python3 --version
pip3 --version
R --version

The minimum versions needed are:

Name Minimum Version
NodeJS 12.22.6
Python3 3.6.9
R 4.1.2

Following Libraries are needed to be installed:

Node

Details mentioned in dependencies field of package.json. Run:

npm install package.json

Python3

Libraries mentioned in requirements.txt. Run:

pip3 install -r requirements.txt

R:

arules package needs to be installed. Run:

R

Once inside the the R shell execute:

install.packages("arules")

Running Instructions

Once the required Frameworks are installed:

Run start.sh

apt install jq
./start.sh

OR

Execute:

To download, extract and convert the datasets:

Make edits to the list of stackexchanges mentioned in config.json and run:

node --max-old-used-space=12288 index.js

Running the Analysis:

Once the dowloading, extraction and convesion to json is complete, run:

cd Scripts
python3 main_.py %%NAME_OF_STACKEXCHANGE_TO_RUN_ANALYSIS_ON%%

For example:

cd Scripts
python3 main_.py crypto.stackexchange.com