This is the code for the supervisor that all crawlers report to.
NOTE: The supervisor can only run in a Google Compute Engine instance.
pip install -r requirements.txt
Create config.py
based on config.py.sample
.
cp config.py.sample config.py
Edit config.py
.
python app.py