/icrwl

Primary LanguagePython

Crawler Project

Requirements

  • Python version >= 3.9

Configurations

Run in terminal:

python -m venv venv

For fish shell ;)

. venv/bin/activate.fish

Install poetry

pip install poetry

Run crawler

Go to crawler catalog

cd crawler

Run crawler commands

scrapy crawl global_vol1

Report results

Jupyterlab

TODO:

  • Add tests/performence tests
  • Go deeper into EDA about links
  • Optimize scrapy