email-jobs-response-apply

💻 (11-Aug-2023) See your Python code do web browsing on your screen with GUI.

Important note:
Installation
Usage
Using docker
iPython Django Shell
API
Admin
Cache react app & view templates
Mail Server
Misc
Contributing
License

Important note:

Before you try to scrape any website, go through its robots.txt file. You can access it via domainname/robots.txt. There, you will see a list of pages allowed and disallowed for scraping. You should not violate any terms of service of any website you scrape.

With selenium we're limited to 10 max ongoing sessions (reference).

I've successfully tested 1000 site crawls in a single process (3 hours, 44 minutes, and 47 seconds).

(4 hours x 1000 sites) * 2 = 2000 sites x 8 hours

2000 sites * 10 parallel sessions = 20, 000 sites

We're able to cover 20, 000 sites / night / machine.

Installation

cp .env.example .env
pip3 install virtualenv && \
  virtualenv env && \
  source env/bin/activate

# chromedriver_mac64
# chromedriver_win32
# See https://chromedriver.storage.googleapis.com
# for drivers list.
wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
sudo mv chromedriver /usr/bin/chromedriver
chromedriver --version

Update config.json with your real credentials.

Usage

XPath cheat sheet.

Update the command at ./management/commands/crawl.py

alias py3="python3"
py3 manage.py crawl
# The app runs at `http://localhost:3000`.

If you still need help installing and running the app check out the readme at https://github.com/kkamara/python-react-boilerplate which is the base system for this python-selenium app.

Using Docker?

alias compose='docker-compose -f local.yml'
compose build
compose up
# Automated runs with Docker:
# compose up --build -d && python3 manage.py crawl

iPython Django Shell

python3 manage.py shell -i ipython

API

python manage.py show_urls

View the api collection here.

Admin

Admin creds are set in ./compose/local/django/start

export DJANGO_SUPERUSER_PASSWORD=secret

python manage.py createsuperuser \
  --username admin_user \
  --email admin@django-app.com \
  --no-input \
  --first_name Admin \
  --last_name User