/sigaa-tcc-scraping-analysis

Sigaa's system scrapping and analysis. This code belongs to an article published in the IEEE transactions on education

Primary LanguageJupyter Notebook

Student's competency analisis: An graph approach

This repository contains some codes used in the article that will be published later.



How it works?

This code uses the following approach:

Image #1

How to run

The project was created and designed under "jupyter researchers" approach. Doing so, you'll need an jupyter interface to open it.

  1. Install python

  2. Install pip (only if you're running on linux):

sudo apt install python3-pip
  1. Install all dependencies
cd src
# if you have the python2 installed too
pip3 install -r requirements.txt
# or
pip install -r requirements.txt

# uninstall
pip3 uninstall -r requirements.txt
  1. You can install the graphviz by the package site or typing:
    • in debian-based linux:

      sudo apt install graphviz
    • or using a choco (windows)

      choco install graphviz
  2. You also need to install the plotly support for jupyter:
# basic packages
pip install plotly==4.14.3
pip install "notebook>=5.3" "ipywidgets>=7.5"
pip install jupyterlab "ipywidgets>=7.5"

# The following commands needs node to be installed:
# JupyterLab renderer support
jupyter labextension install jupyterlab-plotly@4.14.3

# OPTIONAL: Jupyter widgets extension
jupyter labextension install @jupyter-widgets/jupyterlab-manager plotlywidget@4.14.3
  • Optionally, you can create an venv (virtual env):
# install venv
sudo apt install python3-venv
# create a new venv
python3 -m venv pyenv

# Activate the virtual environment
source pyenv/bin/activate
# To deactivate the venv
deactivate
  • You can install jupyter using:
pip3 install jupyterlab

After that, opens the ./src/main.ipynb file and run it.

Documentation

The documentation of modules is inside src/doc/build/html folder.

How to document

The project was developed using Google Docstrings and RST (Reestructured text) guides. You can check here and here.

Sphinx themes:

  • furo (black)
  • read the docs (default)
  • asteroid

Tests

  • You can test the python codes using the script "tests.py":

    # run all the tests
    python src/tests.py
  • You also can use the "vscode testing tool":

    Press: CTRL+SHIFT+P or F1
    and then, type: Tests:
    

Some usefull commands:

# autodocumentation sphinx api (inside source folder)
sphinx-apidoc -o . ../../modules/

# generate html pages for sphinx
make html

# create a new documentation
sphinx-quickstart


# dump mongo
$ mongodump --port 27017 -u USER_NICK -p USER_PASSWORD -d DATABASE --authenticationDatabase admin -o OUTPUT_FOLDER

# restore mongo
$ mongorestore --port 27017 -u USER_NICK -p USER_PASSWORD OUTPUT_FOLDER/

# online restore
mongorestore --uri URI_CONNECTION_STRING OUTPUT_FOLDER/

Why don't use markdown to documentation

See also

To install other themes you must search for a python package equivalent for that theme.

  1. Sphinx themes
  2. Sphinx automodule/autoclass errors
  3. Generating automodules
  4. Sphinx cross references
  5. Sphinx pygments confs
  6. Getting started with sphinx
  7. Getting started with plotly
  8. Mongo mocking (Testing for mongo)